Public Lab Research note


GSoC proposal: v2 API development | Grape | Swagger Specification

by Raounak New Contributor |

Read more: publiclab.org/n/15790


Why do this project?

  • Public Lab is gaining popularity over time. We have our current API which serves the current purpose of client-side consumption.
  • As we have many projects like social media integration which will eventually grow the data, so it would be better if we could come up with a more stable version of our API. It will help Public Lab in terms of growing its popularity as more and more developers start making their apps which will consume Public Lab's API.
  • It would help in making 3rd party app integration which would lead to more apps around Public Lab's vision.
  • We would have a centralized database and can make multiple front-ends (Android, IOS, SPA) based on their specific ideas by consuming the API. Also, the scalability and maintenance of business logic can be easily done, because we will be focusing on a centralized system.

Why Use OpenAPI?

The ability of APIs to describe their own structure is the root of all awesomeness in OpenAPI. Once written, an OpenAPI specification and Swagger tools can drive your API development further in various ways:

  • Design-first users use Swagger Codegen to generate a server stub for your API. The only thing left is to implement the server logic -- and your API is ready to go live!
  • Using Swagger Codegen to generate client libraries for your API in Ruby languages.
  • Using Swagger UI to generate interactive API documentation that lets our users try out the API calls directly in the browser.
  • And more! Check out the open-source and commercial tools that integrate with Swagger.

So I think till now you are convinced that moving on a REST API with the swagger specification is a good option for community. Now lets dive deep into the problem and solution.


About Me


Developer | Designer | Mentor | Speaker

A passionate software developer who uses Ruby to express in code is what defines me.

I am in the field of Web-development form past 3 years and I am loving it. Currently, I am a pursuing my undergraduation in Computer Science and working as a software developer in a company(Namyata). I provide services in Web-development and DevOps. When I am not coding you will probably find me on a football field.


Project description


Abstract

Migration of older API code into the new standard Swagger system, and to make new API endpoints available for the statistical data.

Problem

Things I want to cover up in this project are:

  • Writing new code for API to meet standards of the Swagger system.
  • Making the API with best practices, so that it can be easily maintained and can serve for heavy traffic.
  • Make new API endpoints available for the statistical data, provide at https://publiclab.org/stats.
  • After making this API, we can easily consume the statistical data of public lab for analysis, which further generates many possibilities.

Apart from the above I would also like to handle the DevOps part of this API.


Implementation


Swagger Specification

There are two approaches:

  • A top-down approach where you would use the Swagger Editor to create your Swagger definition and then use the integrated Swagger Codegen tools to generate server implementation.
  • A bottom-up approach where you have an existing REST API for which you want to create a Swagger definition. Either you create the definition manually (using the same Swagger Editor mentioned above), or if you are using one of the supported frameworks (JAX-RS, node.js, etc), you can get the Swagger definition generated automatically for you.

We will be using top-down approach for our new endpoints. These endpoints are mainly of statistical endpoints. These endpoints will be new so it is better to follow swagger specification right from the beginning. For this, there will be the following steps:

1. Logical Design of Statistical Data providing endpoints. In this section first I will brainstorm on how the endpoints should be created i.e, the nesting nature of them and classifying them.

2. Writing them on specification file.

3. Generating Code for the corresponding endpoints with the parameters as specified in particular endpoint.

4. Testing the expected behavior until a satisfactory level has achieved.

5. Documentation and blogs of the features added or updated.

image description

Current State

  1. We are on Swagger version
  2. We are using 'grape-swagger` gem to build our specification file at a route of /api/swagger_doc in JSON format.
  3. From there we are using 'grape-swagger-rails a l o n g w i t h alongwithgrape-swagger-ui` to take that JSON representation of our specification file to make an HTML page for that at routes of /api/docs.

(So we have already had some basic implementation of swagger in our app.)

New State (After Project completion)

1. Newly added API file will also follow the latest openAPI approach

2. I will use the grape's domain-specific language to make the API of the new module like a statistical module.

3. Then by **grape-swagger, **I will make a specification file for it, in JSON format at route '/api/swagger_doc'

4. Moving to the latest version (version 3) for specification.

(How I will implement a new module is explained in next section)

Statistical Integration

Currently, we are showing our statistical data on https://publiclab.org/stats. Here it is rendered inside an HTML view. In this section, I will provide an endpoint to this in JSON response. It will involve the following steps:

1. Categorising the statistical endpoints and nesting them logically.

2. On the basis of the first step new endpoints will be added to existing statistical routes and if required the current statistical routes will be improved accordingly.

3. Specification file will be updated accordingly.

4. The code will be generated from the specification file and will be updated as per our requirements.

5. Testing will be done on the various routes added and updated.

6. If the response is not up to the good level, go to step 1.

image description

image description

Deep dive into making it

1. There will be two files in making this feature work. First one is app/api/stats.rb and the second one will be app/services/stats_service.rb

2. The very first (stats.rb) will have a class inheriting from Grape::API class.

3. Now I will use the features of the grape framework to make the endpoints (described earlier) of statistical usage.

4. While describing the endpoints I will use stats_service.rb to do the heavy lifting work in terms of making a call to the database and fetching the data. The file stats.rb will use the object of stats_service.rb to make calls to the database. This will help in making logic clean.

4. This class will be mounted to the root class where are adding swagger documentation.

5. From there the gem grape-swagger-rails will take the class stats.rb and makes its documentation.

6. I want to change the api URL from {host}/api/{anything} to https://api.{subdomain}/v1/{anything} (This depends on mentor's suggestion)

CSV Support (For Analysist)

At present, there is no support for CSV endpoints. I want to add a functionality in which you get the CSV file having data as per your requirements. This can be included as:

1. Making an endpoint on which all of the available data columns can be seen. [public URL]

--> Now if the user is logged in then proceed further. This authentication will be done on a token basis. This step is required because we can't give out our data to anybody. If needed we can use authorization to limit it to a specific set of people.

2. Select which columns you want in your CSV file and hit the URL (example: /api/v1/csv?{list of params}) with the column names. These name will go in parameters.

3. At the backend, we catch this on a method in our controller (Grape method in this case).

4. From there we will use a custom service which will make the CSV file from it, which is having only those columns.

5. This file will send back as a response to the client.

Token Support for Stateless Nature

I will use the following gems to implement this feature. rack-cors gem is already there in Gemfile.

gem 'active_model_serializers'  
gem 'rack-cors'  
gem 'knock'  
gem 'jwt'

After adding these gems to Gemfile I will do bundle install.

2. Then we will run some generators :

rails generate knock:install

rails generate knock:token_controller user

This will generate the controller user_token_controller.rb and add the required route to our config/routes.rb file.

3. Now we will configure the knock file, as follow:

| | Knock.setup do |config||
| | 
|
| |   # Set how long a login token is valid.|
| |   config.token_lifetime = 1.week|
| |   config.token_signature_algorithm = 'HS256'|
| |   config.token_secret_signature_key = -> { Rails.application.secrets.secret_key_base }|
| |   config.not_found_exception_class_name = 'ActiveRecord::RecordNotFound'|
| |   |
| | end|

4. The controller section of the API will be going in a separate file, which will inherit from Grape::API. And other API files will inherit from it. All the authentication and token related functions will live here and other files can use it by **before_action **helper.

For example, the root file will look like:

# This would give us authentication methods
include Knock::Authenticable

# For authorization also
def authorize_as_admin
  return_unauthorized unless !current_user.nil? && current_user.is_admin?
end

then in any controller or API file we can use:

# If we have set token auth on User model
before_action :authenticate_user, only: [:auth]

5. Since all the code has now been setup for our Token authentication we will start making some routes like /get_token to get token and then start testing.

A POST request to /get_token with credentials will return a JWT token, which client can send in Authorization header in every request to consume API.


Timeline/milestones


Following is the chronological sequence of this project, that I have planned. These are the sequence of screenshots if you want to see the actual one please hit this link.

The very first card in the timeline consists of Preliminary Tasks ( April 1 - May 10 )

image descriptionimage descriptionimage descriptionimage descriptionimage descriptionimage descriptionimage descriptionimage descriptionimage descriptionimage description


Needs


All the things I will need to refer are available on the internet, but I will be needing mentor's point of view on the good practice and security of our API. I will try my best to solve the errors on my own and will need mentor's help only in rare cases.

First-time contribution

I am a first-time contributor to Public Labs. And yes I have gone through the welcome page and have set up the project accordingly. I have done my contribution to OpenSUSE organization before.

I have already started contributing to the following projects:

1. Plot2 (Rails App) : https://github.com/publiclab/plots2/issues/2469

... (more coming)

I am well experienced with the open source contribution and how it works and have contributed to several opensource organization like OpenSUSE. I am providing this information just to clarify that I may not have contributed a lot before in Public Lab, but on my experience and skill set, I can assure you that I can do this project.

I have heard that Public labs welcome new contributors and it is a very enriching experience for them. This is the thing which drives me, as a software contributor in this good.


Experience


Computer Science and Mathematics are my interest right from school day when I was introduced to C++. Since then I started playing around with programming and building some personal and academic projects in C++. I always wonder why there are so many algorithms to do the same work, this thing puts me in the situation where I started finding the difference in between all these algorithms. (Mainly of graphs and link-list).

Then in the college, I got my love, Ruby. Things were so simple as compared to C++, after that, I dig deep into this language and tried things like Meta-programming. I also try my hands on JavaScript, Python, and TypeScript.

After that I started doing web development and it's been 2 years, and I have done the following things:

  • I have worked in 3 startups as a Web developer intern. (Experience letter are on my website)
  • I have worked in fields like UI/UX (While making dashboard in Angular-JS), Backend-development (Rails-5), Browser simulation testing (selenium + Watir), making REST-full JSON API in Rails-5
  • Currently, I am working as a software developer in a company (Namyata), where I am making the full product for them which includes Angular-5 PWA (Lighthouse score: 86) for front-end and Rails-5 JSON API for the back-end.
  • I am handling both production and testing servers of the company which is on Elastic Beanstalk of AWS.

Experience Letter (UI/UX): https://www.dropbox.com/s/q7f9bomwzydv3zi/Aviate_softwares.pdf?dl=0

Experience Letter (Rails Monolithic app): https://www.dropbox.com/s/uq4ae2zfxi83em0/gyandhan.jpeg?dl=0

Experience Letter (Rails Rest API): https://photos.app.goo.gl/Qp83ViprnrvIU6mq2

Academic

I am a University Topper and have studied the core of computing. Subjects includes Fundamental of computing, C, Physics, how the network works and actually what is the problem we are facing now in terms of technology trends and data.

Hackathons

I have been a part of many hackathons including regional hackathons of various universities(Hack@DTU, Hack@NSIT and many more) in nearby cities and also international ones like AngleHack. One of my driving force to learn new technologies has come from there. Those sleepless nights with code have really pushed me to make the best software possible.

Mentor

  1. I was the coach at Rails Girls workshop held in Pune, on 21st January 2018.
  2. I am currently a coach of 3 teams in Rails Girls Summer of Code.

Speaker

I build maintainable and market stable products, and take the responsibility of the project/product I am working on. For your satisfaction, you can check the feedbacks of my clients on my Personal website or on my LinkedIn. (link above).

I have set-up the development environment on my local machine. I have also worked on following issues 1. https://github.com/publiclab/plots2/issues/2469

Pull request submitted: 1. https://github.com/publiclab/plots2/pull/2500 2. https://github.com/publiclab/image-sequencer/pull/195

So I have experience and setup both plot2 and image-sequencer. On those basis I can say I have familiar with the development phase of Public Lab.


Teamwork and self-motivation


I have worked in a team several times:

  • In my first internship, it was a team of two. Me and senior developer, in that I learned the importance of regular update from both the sides (Junior and senior developer).
  • In my second internship, it was a team of 4 developers and I learned what other developers (Data science interns) and clients actually need from web developers. I understand the importance of clean and to the point git commit messages, and actually how branches work in git and GitHub. I also learned more about version control when I started contributing in OpenSource.
  • In my current company I am leading a team of 3 members in which I handle design/front-end interns, here I understand that things changes very fast, and regular updates and git commit history is much important than we think.

So keeping the above experience in mind I think, I can work in a team without disturbing the mentor for common bugs and simple things which I can google. Git commits messages are to the point and relevant so that it won't create a problem in future

Self Motivation

Going into various hackathons, attending conferences and being a mentor, it is my responsibility to teach the right thing. And for the same, I work hard in order to understand the things and deliver them in the correct manner.

  • I am a big fan of a book "Software developer lifecycle manual - by John Sonmez", this is one of my great self-motivation boosters.
  • I have done 10+ online courses (you can see certificates on Linkedln), to keep me up to the standards time to time.
  • I have worked remotely many times. So I have the experience to keep up the things even when they do not work as we expect.


Passion


After some alarming dangers to our environment people around the world started doing something to improve the environmental conditions around them. But it didn't have that much effect as they expected because people around the world didn't communicate with each other and lack of support will bring down anybody.

I want these people to connect and support each other so that a major change could take place.

Public Lab is what is what we all are looking for. Public Lab can help me in to join this league of DIY people. As I am a software developer I am ready to give them full software support as per their needs. I am also impressed by the different fields of people in the community.

I strongly believe that with people having this much of experience in different fields, we can definitely build something awesome, and I want to be a part of that.


Audience


I think the project will improve the performance of the API, which will lead to the better serving of the clients. With the statistical data endpoints, we can make apps to better monitor the data and to improve data analysis on some low power devises. Also, we can make this data available for 3rd party integration and people from around the world can access it for their own usage.

I will keep my code as per the industry standards i.e more modular, more test driven so that there will be no problem in further development.


Commitment


I believe that this work needs attention and hard-work equivalent to a full-time job and I am ready to deliver that. My main motive is to complete the work before time and also get some time for feedback and production bug fixing.


What to expect after completion


  1. A very well written code in terms of modularity, test-driven development, best-practice, no robocop offense.
  2. Well explained documentation.
  3. Example Usage show in the documentation.
  4. Video tutorial on how to consume application on various front-end frameworks.
  5. Blogs at various steps of development.

And an increase in clients consuming our API for their products/projects.

Can't wait to start working on this.

Thank you

Raounak Sharma



software gsoc grapes soc gsoc-2018 soc-2018 soc-2018-proposals rest-api rails-api swagger soc-2018-api

response:13975


19 Comments

Hi @raounak thanks for your proposal. It will be more appreciable If you add more details in your proposal means different stages of work and how you'd break up these tasks into a checklist where you could submit completed work each week, that would be great. You'll end up with somewhere around 12-15 weeks, isn't that right?

Thank you!

Is this a question? Click here to post it to the Questions page.


Hello @mkashyap354, thanks for replying back. I am working on that part (Timeline). As my university examination are going on (till 28th February) that's why it is taking time. But I will submit it in a week as I am very interested to work on this project.

Thank you


Ok @raounak , no problem


Hey @mkashyap354

I have started working on making the timeline. I am having some doubts regarding code base and features we need to add. What would be the good way to ask. Can I mail them you?

Thanks

Is this a question? Click here to post it to the Questions page.


Hi @raunak , good to see you again. Yes you can mail me but if you ask your doughts on our channel in gitter or IRC then it will be more appropriate. On gitter, name of our channel is publiclab


Hi, @Raounak - once you have a chance, it would be great to see a small contribution to ensure you know our workflow -- and it's very valuable experience for you as well!

https://publiclab.github.io/community-toolbox/#r=all

We're happy to help you take the first step!

You may also want to coordinate with @mridulnagpal about the geographic APIs we are discussing in that proposal -- maybe there's opportunity to team up! https://publiclab.org/notes/mridulnagpal/01-22-2018/leaflet-blurred-location


Hello @warren I have asked for this issue as a first time contribution, and may be give a patch for it by today itself. And for the other thing I saw that proposal by @mridulnagpal, and I think it will be great to merge this two proposals into one and make a useful API. But I haven't heard of team-up in GSoC proposal. Is it possible that we two can gave one proposal as a team?

Just a doubt.

Is this a question? Click here to post it to the Questions page.


@warren, I did my first PR. There was no issue left unsigned in plot2, so I did it in image-sequencer. Link: https://github.com/publiclab/image-sequencer/pull/195

Now looking for more issues in plot2 or any public lab projects, as you said there we can solve issues in Editor project. It's fun and learning contributing in open-source.


@warren, @liz, @mkashyap354

I have made changes in my proposal, please review it and share your point of view. If there is any specific change you want or have any suggestion about it, then please share, I will be more than happy. Please have a look and tell.


Hi! Thanks, this is looking good, and i'll just reply to things in a list here --

  1. I like the background info on Swagger and OpenAPI -- can you describe how this relates to the current state of the API? And perhaps make reference to the API docs at https://github.com/publiclab/plots2/blob/master/doc/API.md ?
  2. I love your mention of the https://publiclab.org/stats pages as a target for API development. This is a long-standing goal, and I want to specifically highlight CSV download so that people working on analyzing community work on Public Lab (like @liz) can download and graph the data efficiently. The API doesn't currently offer CSV and only some RSS format endpoints, and I'm not sure how this relates to Swagger conventions. Can you elaborate? Thanks!
  3. Your diagrams are really nice! I wonder if we could elaborate a bit on the relationship between the API endpoints in files like this: https://github.com/publiclab/plots2/blob/master/app/api/srch/search.rb and the Services in in files like this: https://github.com/publiclab/plots2/blob/master/app/services/typeahead_service.rb -- the former are functional-tested, while the latter are unit-tested (or should be!) and can be used elsewhere in the application. How might this structure be used to implement a more consistent Stats system -- perhaps a statistics_service.rb file?
  4. finally i love the detail in your proposal on things like tokens, versioning, caching. We see pretty high load on the /stats page queries, so that may be something we need to control for. We also already have a preliminary token system created by @ryzokuken -- perhaps that section of your proposal could talk more about how that can be integrated into the API, or how more token-based functions could be built out from that start?

Thanks a lot! Don't forget to take a look at related proposals by @milaaraujo, @stefannibrasil, @rishabh07, and @sukhbir -- and reach out to talk with them about their proposals too! #soc-2018-api

Is this a question? Click here to post it to the Questions page.


yeah i agree with @warren, and I also recommend you to look more issues or make contributions in our organization which helps us to know about your workflow.


Hello @warren I have tried to answer your first 3 points you made.

  1. I added the current state and future state of swagger in our project.
  2. I added a CSV support for statistical data and how I am planning to implement it. (@liz)
  3. I explained how api/srch/* files are connected to services file by taking the example of stats feature I will implement. (If need I can explain more. For now, you can think it of as services in Angular-4/5 and how they work for a component).
  4. For token based auth feature I am researching more on best practice and will update in a day.

And yes @mkashyap354 I am working on an issue https://github.com/publiclab/plots2/issues/2469 in plot2 and before that I have a PR merged in ImageSequencer. I will try to solve more is time allow

Thanks to all, and I am looking forward to do this. Let me know your point of view on this.


@warren, @mkashyap354, @liz, @vidit I have updated my proposal. Please have a look and give feedback.

I have also shared it from summer of code website. Have a look at there also.


@raounak this is a great proposal. I learnt a lot from your proposal thanks.


Thank you @bansal_sidharth2996 for kind words.


Hope others will also like this.


Nice work @Raounak , I like the way you made the timeline.


Hi @Raounak !

Really amazing timeline ! And I learned a lot about 'grape-swagger` gem . And the flowcharts are really cool . Found nothing to add here :)

Thanks !


Yes, this is an extremely thorough proposal. I have nothing to add, great work!


You must be logged in to comment.