# GSoC proposal: Email integration Project

by namangupta |

(This template is for students applying to summer code programs with Public Lab. Use this link to start writing a post: https://publiclab.org/post?n=899&tags=soc,soc-2018,soc-2018-proposals You can delete this line once you've started filling it out.)

Name: Naman Gupta

Affiliation: Guru Gobind Singh Indraprastha University, New Delhi, India-110078

Location: Delhi, India-110034

Github: namangupta01

Email: 01namangupta@gmail.com

Project: Email Integration Project

## Project description

Email integration project

### Abstract/summary:

Implementing delayed background jobs, cron jobs for sending daily/weakly/monthly/ trending email digest to the subscribers.

### Introduction

The current implementation of Email System in PLOTS2 doesn't implement Active Jobs i.e queue feature because it is currently using rails 4.1. And rails 4.1 doesn't implement Active Jobs on its own. We have to ways to resolve this either include activejob gem or upgrade to rails 4.2 (we will go with this). So currently, mails are sended synchronously. And if the sending of mail is failed due to some reason or we don't want to send the mail at the time of request/response or if we want that a mail should be sent after (1 day or any specific period of time) we can't implement it now.

Sometimes, we have to run jobs at particular intervals to, say, backup logs, send emails, etc. In rails application we can schedule such tasks using the whenever gem.

For integrating cron jobs with the plots2 we can implement whenever gem which will be very useful in sending daily "batch digest" email to the subscribers. We can specify the time when the email should be sent. More details on implementation is in Implementation part.

Another feature is the reply-by-email feature using mailman gem which currently have three types of receivers: Standard Input, Maildir, and POP3. POP3 is used most often.

Another feature that we can implement is of reply by tweet which is very convenient and easy for the user to do.

### Project Goals

During this GSoC season I would deliver:

• Including Active Jobs in project.
• Implementing whenever gem to work with Active Jobs to perform delayed jobs.
• Implementing mailman gem which will be used in reply-to-comment via email feature.
• Implementing reply-to-comment via email feature.
• a user interface for managing digest settings and email notification settings.

### Implementation

3.1 Including Active Jobs

Currently, plots2 is using rails 4.1 and rails 4.1 don't implements Active Jobs so it should be updated to rails 4.2 so that we can use delayed jobs with the help of Active Jobs. On which i am currently working.

3.2 Implementing whenever gem

For implementing cron jobs with our application we can use whenever gem. The steps for implementing this gem are specified below :

1. Include whenever gem in Gemfile

2. Type command wheneverize . in the root folder of the application which will create an initial config/schedule.rb

3. After typing above command also type command whenever which will create a file schedule.rb

4. Now edit the schedule.rb file to add cron jobs. The best things about whenever gem is that it makes it easy to write cron jobs in simple ruby language.

• For Example:
every 1.day, at: '4:30 am' do
runner "MyMailer.send_daily_digest.deliver_later"
end

every 1.day, at: '4:30 am' do
end

every 3.hours do # 1.minute 1.day 1.week 1.month 1.year is also supported
runner "MyModel.some_process"
command "/usr/bin/my_great_command"
end


5. Now run command whenever --update-crontab update the schedule.rb file and to enjoy cron jobs.

3.3 Implementing mailman gem

User will receive an email whenever he is being mentioned in a comment or whenever someone is commenting in the node that the user is following. And if he wants to comment or want to reply the comment, this feature would make it very easy for him because he don't have to open the website and then comment. He can directly reply via email using this feature.

This feature can be implemented with the help of mailman gem. Mailman is an incoming mail processing micro framework that support POP3 and Maildir support, that works with rails "out of the box".

Implementation steps:

1. Add mailman gem in gemfile and then do bundle install.

2. Now we need to add a script in script/mailman_server. This script will configure the mailman setting.

3. Currently, there are three types of receivers in mailman: Standard Input, Maildir and POP3. POP3 is very commonly used so we will use POP3 receiver.

4. POP3 receiver is enabled when the Mailman.config.pop3 hash is set in script/mailman_server. This will poll every minute by default. Polling time can also be changed. After new messages are processed, they will be deleted from the server by default but if we want to save them we can also save them.

5. We have rails environment present in script/mailman_server which means we can access rails models there.

6. script/mailman_server also contains a module which will run when a mail is received. As specified above we can use our models there because rails environment is there. Here's the screenshot of how the whole script/mailman_server looks like.

7. Now receive_mail function in comment model will contain the code for finding node and Sender email, so that the comment can be created. So the question comes, how we will decode the node Id and how we will get the sender's email. We will get sender's email from the From of the mail which contains the sender's email address and we will get the node id from the subject of the email. For example, subject would be like publiclab node #1; here 1 is the node id, so we will use regex to find out the node id and then add comment to the node. This is exactly same that happens in Github where issue or pull request id is used in subject.

8. Add a icon when displaying comment to differentiate between normal comment and reply by email comment.

3.3 Implementing feature reply by tweet

Reply by tweet is an amazing feature that would led users to reply to any node via tweet. Currently, we are posting every node at http://twitter.com/publiclab (using IFTTT.com). We can easily do it by using Twitter Webhooks.

**
**

Implementation Steps:

1. We will use Twitter **Web-hooks **Api ( Webhook api ) to get all the things happening on the public lab twitter account whether it is tweets, reply, mentions and lots more.

2. Using web hooks, we will get all the data in real time, no polling is to be done using any gem, in polling permanent connection is made between twitter and server using sockets by handshaking of our server and twitter.

3. As we will get all the activities happening on twitter including reply, the next thing is to identify the node.

4. Now, all we need to find out to which node does the tweet belongs. This can be done in two ways. First way is creating a new model via migration to save the node vs key i.e in ift.tt/2G6IK69, 2G6IK69 is the key which we will store in reference to the tweet and we will store node_id in reference to the node. Or, secondly we can tweet an tweet_id_ along with the tweet and save tweet_id and node_id in our database.

5. As we will receive reply using web hooks, we just have to identify the tweet to which the reply belongs which can easily be find by the service provide by twitter.

7. We can also add a icon to differentiate between reply-by-tweet comment and normal and reply-by-tweet comment.

### Timeline/Milestones

Community Bonding Period [23 April to 14 May] -> Get to know team and the mentors and also start discussing, analyzing the areas and discussing the work with mentors.

[14 May to 20 May] -> I have my exams from 14 may to 22 may so my working frequency will be less in this period (from 21 may to 22 may).

[21 May to 27 May] -> I will start working on including active jobs so that further work of performing delayed jobs can be done.

[28 May to 3 June] -> I will start working on the feature to include whenever gem so that we can perform weekly/email/monthly/ trending email digest.

[4 June to 10 June] -> I will start working on implementing the feature to implement mailman gem so that we have an reply by email feature.

Phase one evaluation [11 June to 17 June] -> Working on creating Mailers for digest.

[18 June to 24 June] -> Start writing tests of what is done till now.

[25 June to 1 July] -> Integrate twitter webhooks to get realtime activity from twitter of what happening on the publiclab twitter account.

[2 July to 15 July] -> I will start making model and writing documentations and working on logic to add comment on a particular node.

[16 July to 22 July] -> Finally working on improving code structure and working on view of how the comment looks.

[23 July to 29 July] -> Writing tests of the code written up till now.

[30 July to 6 August] -> Finally take a week to scrub code, improve documentation, etc.

### Experience

I have been working on the the Web Technology from more than 1.5 year. And have developed many applications on rails as well as on other technology.

I have also worked as Tech Head in Google Developers Group - BVCOE which is a technical community and also took various workshops and sessions as mentor on different technology.

I have also worked as Teaching Assistant for Web Development Course (which is a Ruby On Rails course) in Coding Ninjas (www.codingninjas.in) where my job was to mentor the student and help them in their Web Projects.

I am currently working as Web Developer Intern in Edusquare and working on building their online test platform for preparing for prestigious Engineering Entrance Exams in India. These platforms are build using various technology -- Ruby On Rails, Python, Flask, AngularJs and Backbone Js.

### TeamWork

I have been actively volunteering in various community where i have showed my skills of teamwork and hard work. I am currently volunteering in Google Developers Community and i have also organised hackathon in my college and worked as one of the organiser.

### Open Source Contribution

I have been working on the open source projects from the past eight months. I have contributed to organisations like Public Labs and OpenSUSE.

### Passion

The things that interest me the most about public lab is the science and environments because i am very passionate and have very deep interest in science particularly in physics --understanding how the law works in nature.

### Audience

This project will make the publiclab user experience good whether it is reply by email feature or getting daily/weekely/ monthly/treading email digest to the subscribers.

### Commitment

Yes, i do understand this is a serious commitment and i will give my full dedication and commitment to this project .

software gsoc soc gsoc-2018 soc-2018 soc-2018-proposals soc-2018-email gsoc-2018-final

response:13975

Wheeee! A third proposal for email integration! I'm going to re-iterate what I wrote to @gauravano and @vidit -- this is a highest-priority project for us, so we'd love to see multiple people tackling this from different sides. It's not a problem if multiple people are accepted; we will have to break the broader project down into parts so that it's possible for multiple people to work on at the same time.

For all three of you, do look at proposals by others doing Rails 5 upgrade work: @souravirus, and @meghana-07 have both proposed this. #15826 #15814 There may be some cooperation and work sharing on the Rails 4.2 upgrade which ActiveJob requires -- even before the summer begins!

Perhaps with three people interested, my comments about how to break it up could be taken even further -- discuss! I also proposed a possible "reply by tweet" concept on @gauravano's post, so maybe that's another area where a third person could get into things.

Awesome, please read one anothers' posts and provide each other with some feedback! And don't worry about overlapping -- as i said before, it's a powerful motivator to break things up into smaller parts so that multiple people can work in parallel on different parts of the problem. Thanks!!!

@warren i would love to help in upgrading to rails 4.2 and even i am currently working on this. And as you said about the reply-by-tweet concept on @gauravano's i am happy to work on this. I will update the proposal regarding this concept soon.

And @warren what do you think of having an email creating editor for the admin to send emails to the users ?

Is this a question? Click here to post it to the Questions page.

@warren i just have one doubt in reply-by-tweet concept. If this is going to be used for replying to comments then how will the user will do tweet in reference to particular comment? We have to give some reference in the tweet so that the comment is identified and rest of the things can be handled by cron jobs using whenever gem.

Is this a question? Click here to post it to the Questions page.

The idea with "reply by tweet" is that we automatically tweet out each note -- this we do already, at http://twitter.com/publiclab (using IFTTT.com) -- so any replies to that tweet would get copied back to the original note.

You're right -- the reference back may be tough. We could:

1. follow the link in the tweet to resolve the correct node to comment on
2. post the tweets in a different way and save the tweet ID, and perhaps save that in a tag like tweet:___ID___ on the node

i'm sure there are other ways too!

email creating editor for the admin to send emails to the users

This is interesting -- what about something similar, where we have a way to notify groups of people, like how we do on GitHub like @publiclab/reviewers -- a "group" handle. On plots2, we could do: @g/partsandcrafts -- to notify people who have the profile tag partsandcrafts. We may also want to ensure that only moderators can do this, or it'll create a lot of email flying around!

Let's make an issue for this last idea and add it to the list of email features. It's not high-priority but could be pretty easy to implement, given profile tagging and the existing email templates.

@namangupta as the timeline is near can you please update the remaining sections so that the potential mentors can review it ??

Is this a question? Click here to post it to the Questions page.

Hi @namangupta !

I have some experience with webhooks and a couple of things to add here :

1.) If you plan to use Webhooks than why not then create a webhook to post tweets real-time , instead of using IFTTT ? Does this makes sense ?

2.) PublicLab has all the URL's and API's open , i.e anyone can access them . The URL used for Webhook provided by PublicLab will be open , right ?

• For a big organization like PublicLab , do you think it would be a safe option ? Especially what if someone gains access to this URL and sends TB's of data here , which will result in spam data and ultimately server crashes . Our website will be flooded with bad data in this case .

• This may be a extreme case but just trying to be cautious here ! You may be aware of the recent DDoS attack on Github , where TB's of data was transmitted (it is the biggest attack till date) .

3.) I remember that it took me like atleast 2-3 weeks to get my account active for Webhooks . Is the approval time decreased now ?

4.) If in production , there is any error in our application, then the tweets that will be posted in that timeframe will be lost till the error is resolved (bcoz the twitter webhook will not pay attention) , makes sense ?

5.) How to you plan to debug ?

6.) The key (generated by IFTTT) comes after the tweet gets published . right ? So how will you add Key vs Node entry in database ?

7.) And you may have answer to this but just to add here - how do you plan on getting replies of a tweet as Twitter API does not directly give this directly ?

The Email part looks perfect to me and I learnt a lot :) Thanks !

I hope this helps !

Thanks so much :)

Is this a question? Click here to post it to the Questions page.

I just realized i asked too many questions here ! Sorry about that !

As @warren said here , you do not have to answer them before the deadline .

So Sorry again ! Thanks !

@sagarpreet no problem. I am glad to see that you are so much interested in my proposal. Thanks :)

Great !

You must be logged in to comment.