Public Lab Research note


GSoC proposal 2021 : Translation System Refinement

by ajitmujumdar25999 | April 05, 2021 21:17 05 Apr 21:17 | #26143 | #26143


About me

Name: Ajit Mujumdar

Email:ajitmujumdar25999@gmail.com

Github:https://github.com/imajit

Affiliation:National Institute of Technology, Karnataka

Location: Gujarat, India

Project description

Setting up a workflow for importing new translations, recruiting new translators along with fixing HTML parsing and UI shortcomings by fixing CSS and helper-function related problems.

Abstract:

Public Lab has a great feature to view some contents of the site in different languages. We are currently using rails-I18n gem with translations arranged in YAML files and loaded into views using the translation helper function. I feel it is a great initiative to spread knowledge with people of various native languages.

Problem:

What problem(s) does your project solve?

Our internationalization (i18n) system exists but lacks a consistent workflow for importing new translations, as well as suffering from some HTML parsing and UI shortcomings.

Project Goals and its implementation

Break your project up into small projects -- one per week!

Before discussing the possible ways to improve the existing workflow let's analyze the existing one:

  • For a new user unaware of the translation system there is no pop-up, notification, or anything which helps him/her know about this.
  • Let's say he/she finds the Translation wiki follows the step in it and becomes a Translation team member, but now all over the site he/she starts seeing globe icons redirecting to the Transifex team page with so many icons and buttons out of style it affects his/her experience.
  • Let's say he/she wants to interact with the translation community share ideas, learn something new there is no community discussion platform that has such discussions as well as information on how to resolve commonly faced issues.
  • Let's say he/she wants to add new translations to the site but is not familiar with Github, it will be difficult for him/her to manually merge changes. Transifex has an automated setup to merge translation but with no notes/wikis about how to merge changes there.
  • Once the community starts working, we need to keep them engaged with new tasks, features to make the experience better, and ways to appreciate the contributors.

These are some of the roadblocks that I feel we can work on to refine the Translation system.

We can improve the existing workflow by:

1.Translation Dashboard:

Translation of website infrastructure is done by integrating Public Lab with Transifex. With more and more strings and languages being added, we need translators to help us translate and review translations. To streamline the process of translation, we can set up a translation community where users willing to help can join in, interact and contribute to the organization.

Currently, the pop-up about missing translation redirects users to the Transifex team's page.

Functionalities - To set up such a team we need these functionalities in Dashboard for recruiting and maintaining:

  • Spread awareness about this initiative
  • Make on-boarding to the team as simple as possible
  • Discuss ideas, resolve queries, and record suggestions and feedback for improvement.
  • Appreciate contributors for their efforts and keep contributors updated and engaged.

A brief overview of the recruitment process is :

Recruitment Process


Reusing existing infrastructure:

We currently have some really great resources in the form of wikis and tags that contain information about all the phases of the Translation system, what we need to do is set a workflow using these resources and align them in such a way that they are more intuitive and accessible.

I can understand that it would be really great if instead of building a completely new Dashboard, we use the existing Wikis, notes, and tags to streamline the Translator Recruitment. The existing setup has:

  • A Translation wiki that has nearly all the information one needs to know about Translations.
  • A Translation tag where one can follow all the discussions and resources about Translation in Public Lab.

This already solves two issues a Dashboard is needed for. We can add some more information about Translation Tag and also create a new Research note dedicated to discussions, suggestions, and queries about Translation System. This can help us avoid maintaining a new system by using the existing resources.

Just to map out how all the functionality discussed earlier can be setup :

  • Notifications and resources: Translation Wiki Page and Translation tag which currently exists with some minor additions are good for this purpose. We also have globe-icons to redirect to Transifex team, we can redirect it to Translation Wiki which acts as Dashboard where user can find all the resources.We can also add a notification message in sidebar for users not having
'translation-helper'

in profile-tag something like

to tell them about Translation System.This can be visible when one switches from English to other Language.

  • Recruiting new translators: We use Translation Wiki as a Dashboard.The wiki page has a step-by-step article that shows how one can be a translator by signing up to Transifex. So we can ask interested users to follow these steps, send a request to join Transifex Translation team on Transifex site itself and subscribe to Translation Tag to get all updates.
  • Discussion forum: We can create a research note with the name "Translation System Discussions" where we have some basic information about Translation and in the comment section of this note can be used as a discussion forum.Users can give suggestions, resolve queries and discuss other issues.
  • Statistics about Translation: Transifex already has an interactive setup to show statistics about users and translation progress. Optionally we want to have something similar on the Public Lab site we can set up API calls to get data using Transifex API.
  • Community Engagement: We can keep a list of top contributors, new contributors in the Translation Wiki and users can share their experience in form of research notes with a Translation tag. Users can get updates by subscribing to Translation Tag.
  • (optional) To avoid spamming of Public Lab Transifex account we can have a simple form fill where users can share their Email-id and we can send an invitation. To invite new collaborators we can just collect the user's preferred email ID from the form and then use Transifex invite feature to send an invitation mail. Admins can copy the email and send invite. Removing members will also have a similar workflow.

Public Lab currently has a github-integration with Transifex to pull new translations and push new strings for translation to Transifex. We can extend this GitHub integration by adding an API integration with Transifex. Some API calls such as ones related to Statistics and Team Management can be useful in Dashboard.

Translation Wiki as Dashboard

Translation Tag page to access notes,questions,wikis and reach out to other team members.
Pros of having a dashboard:

  • We can have one place to find all resources, check the status of the project, and recruit new translators
  • It can be a very smooth and interactive experience for new translators.

Cons:

  • Currently one-click redirects us to Transifex so users familiar with Transifex might feel it more time consuming to apply on-site first and then use the invite link to join the team

2. FTOs issues workflow:

Many FTO issues were added as part of the Translation System Project where new contributors could understand the workflow by changing the translation function


t 'text to be translated'

to a translation helper function to translate word in case its translation exists else prompt message about missing translation.

Some of these issues are:

  • As most of the contributors are new to the organization many might just change the required files but not test the changes in the local environment before pushing changes.
  • There is no footer in local env and dropdown to change language is part of the footer, so to change the language one needs to use http://localhost:3000/change_locale/es to switch language, and then changes are visible. Many don't know about this so don't test UI changes issue this is one such issue where it would be great if contributors had tested changes locally issue could be avoided.

To streamline this process we can have a GitHub bot that finds 'basic translation' kind of tag on pull requests and issues, and adds a review comment which contains:

  • Best practices for changing files.
  • How to test changes locally
  • To attach screenshots of changes in PR
  • Explaining why we are adding these changes

We can add something similar to this in the community toolbox where all issues under translation can be shown along with comments stated above as a description to the section.

Write tests for these changes to make sure the UI does not break on adding translation helper.

Adding these would make reviewing and merging these issues very efficient.

In the community box section, we can appreciate top contributors for their efforts.


3. Improving Translation Helper:

The current translation helper does not work correctly in some cases. We need to improve the helper to handle complex HTML parsing and strings that have escape characters.

This is the existing helper function

Some fixes I can suggest for now are :

  • check for escape characters and other such corner cases in the helper function itself.
  • the function can be to return safe_join or sanitize input instead of raw can help fixing issues with escape characters.
  • Translation Sandbox - To test the js and ruby helper function on various parts of the site we can keep a dedicated page in the local environment which acts as a sandbox and has all the possible UI features where the helper function is used. This can be very helpful for contributors to check changes made in HTML, js, or ruby parts.
  • Similar to a Ruby-based helper function we can set a js-based helper function to check if the logic when ported from ruby to js works better and resolves parsing issues.

But I feel this issue can be corrected by a bit of trial and error so I feel it can be solved once we start working on it.


4. Improving CSS:

Currently, if the translation of a word is missing then we have a globe icon that shows translation missing pop-up and redirects users to Transifex teams page. This icon sometimes breaks the CSS layout and appears in a lot of places.

We can add these changes to improve the CSS:

  1. The globe icons are only visible to users having

'translation-helper'

tag in their profile tag so for users not having that on switching to a language other than English we can show a notification to help with translation.

This is a sample notification that we can add on top of the page or in the side-bar which can redirect the user to the translation dashboard.

2. We use a dropdown to change the language so we can use icon instead of to show where translations are missing.

We can also change the pop-up message to something shorter.

Current UI

After changes.

3. Too many globe icons are rendered on screen, we can clean the UI a bit by reducing the number of icons to a fixed value for each page.

Current UI

UI with a lesser number of icons.

4.We can fix buttons and dropdown styling such that icons stay inline with the button-text.

These are just some suggestions to improve the UI, we can do alterations to this later based on discussions.


Stretch Goals:

Translation for Notes:

To translate research notes we can add an option for translators to translate text in a portal similar to the one below where on one side there is an original post and on another side we can have the translated post if anyone has initially worked on it or else a fresh text area where translator can write the translation.

Once the translation is complete, he/she can send it for review. Once the text is reviewed based on the language selected or region of the browser we can show the option to view translated notes for users.

Pros of having self translation-review system :

  • To send and receive translation to Transifex we need to share it in a .yml file but it is not possible with notes that are added on a day to basis.
  • This can add value to the translation community with people and keeping the translation team engaged with new notes regularly.

Cons :

  • Need to assign reviewers for translations
  • Need to add more functionality to existing architecture to store data in more than one language
  • Need detailed discussions about which wikis to translate, how reviewing will take place, how wikis having more than one language be shown to users.

I really want to work on this as I feel having Wikis in multiple languages can help people understand them much better but as it requires a lot of planning and discussion, I would like to discuss its feasibility during the community bonding period.


Timeline/Milestones:

Time-period Tasks
Community Bonding Period
  • Discuss implementation details with mentors

  • Decide Dashboard setup
  • Setup project goals 

  • Prioritize critical issues

Phase 1
Week 1 (June 7, 2021 - June 14, 2021)
  • Setting up Translation Dashboard

  • Adding necessary resources and links for translation

Week 2 (June 14, 2021 - June 21, 2021)
  • Setting up Transifex API

  • Writing tests for new functionalities.

Week 3 (June 21, 2021 - June 28, 2021)
  • Completion of Dashboard

  • Setting up FTO workflow

Week 4 (June 28, 2021 - July 5, 2021)
  • Adding gitbot for translation-based FTO issues.

  • Adding Translation Section to community toolbox

  • Adding unit tests

Week 5 (July 5, 2021 - July 12, 2021)
  • Testing changes

  • reviewing and feedback on changes.

  • Documentation

  • code clean-up

Evaluation of Phase 1
Phase 2
Week 6 (July 12, 2021 - July 19, 2021)
  • Setting up Sandbox for UI changes due translation based issues.

  • Adding tests for sandbox
Week 7 (July 19, 2021 - July 26, 2021)
  • Adding JS-based helper

  • Improving translation helper function

Week 8 (July 26, 2021 - August 2, 2021)
  • Improving CSS

  • Correcting issues with complex HTML parsing 

Week 9 (August 2, 2021 - August 9, 2021)
  • Reviewing changes

  • Writing Tests

  • Documentation and code clean up

Week 10 (August 9, 2021 - August 16, 2021)
  • Buffer period for reviewing and adding suggested changes

  • ***Stretch goals

Evaluation of Phase 2
Week 11 (August 16, 2021 - August 23, 2021) Code Submission and Final Evaluations
** * - Implementation based on time constraints

Needs

I will require guidance from my mentors, suggestions, and feedback from all the members of Public lab in the GSoC weeks.


Contributions

I have been contributing to public labs since January 2020, I have contributed to issues involving HTML, CSS, js, and rails. I'm familiar with the tech stack and also interacted with the mentors while solving the issues. I have tried to actively participate in discussions and give my suggestions. I'm also been active on public forums in helping out new contributors and fixing issues.

Issues

PR

Comments


Experience

Tell us how you've learned about writing software; what languages you've been learning, if you've worked on other projects, links to GitHub or other code repositories or samples.

I'm a final year student at NITK, Surathkal and I have been in development for around 3 years.

  • I have done a 10 weeks summer internship at Intuit India where I worked on database migration and data visualization and analysis.
  • I was one of the winners at Walmart Hackathon 2020 and Smart India Internal Hackathon 2020
  • I have been at the finals of Smart India Hackathon 2019 and 2020.
  • I have worked on a startup related to e-commerce and advertisement.
  • I have worked on development-based projects as part of course projects and college tech-fests.
  • I have decent knowledge about js, rails, and git and I'm good at OOPs based concepts.

Teamwork

__I have participated in various team hackathons at college as well as national level. I have also worked as a team on video captioning-based research work. In Public Lab, I have got a chance to learn open-source practices and work with some really helpful people especially@jwarren,@cess,@nstjean,@Uzay-G. It has been a great journey contributing to Public Lab and being a part of such an amazing community is a special feeling.


Passion

Public Lab strives to solve environmental issues and is a community platform for innovative ideas and research notes related environment.I would love to work with this organisation and contribute to the FOSS community and solve real-world issues,with refinement to translation system I hope to get more people to share their knowledge and spread it across in various languages.


Audience

This project will help in recruiting new translators for Public Lab and help new translators understand the workflow and add missing translations.


Commitment

Yes, I fully understand that this is a serious commitment and I can 25-30 hours weekly for the completion of this project.


5 Comments

I'm still working on some parts of the proposal. It would be great if @warren @cess @ruthnwaiganjo and other mentors and contributors at Public Lab can share your views about this draft proposal

Reply to this comment...


Thanks @ajitmujumdar25999, for this proposal! I'll leave some comments just in a list:

  • I like your thinking on infrastructure (like the spam dashboard) to manage and coordinate the community of translators + reviewers, and I totally appreciate that more of a 'system' would help! I wonder, though, if we could use the topic/group infrastructure of PublicLab.org itself rather than build a new dashboard. For example, https://publiclab.org/wiki/translation lists some activities, and we could think about recruiting people to follow the Translation /topic/ at https://publiclab.org/tag/translation to get involved. We could make the wiki page the "dashboard" perhaps? I'm just thinking about how to be judicious in creating new systems that would have to be maintained as well.
  • I love that you've mapped out the recruitment flows. It'll be great to think about where the "roadblocks" are and to ensure for example that people who don't have GitHub accounts are supported. You can see a bit of this happening already at https://github.com/publiclab/plots2/issues/9481 but most translators/reviewers may not even be familiar with what GitHub is!
  • safe_join is a very cool idea! I do notice that it doesn't actually close stray HTML tags, but it definitely would escape them if they're not valid. That said, i agree it's a good short-term improvement that would make things fail a little more gracefully.
  • finally, i think a set of robust tests for inserting the helper icon and tooltip into different HTML scenarios would help us diagnose. We might have a test page with a variety of HTML content in it, which we attempt to run the translation helper on, and to write a test around this? At least so we can visually review how each scenario renders. This would also make it easier to confirm that a JS vs. a Ruby helper were both functioning as expected with a variety of challenges!

Thanks a lot, @ajitmujumdar25999 -- much appreciated!!!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I like your thinking on infrastructure (like the spam dashboard) to manage and coordinate the community of translators + reviewers, and I totally appreciate that more of a 'system' would help! I wonder, though, if we could use the topic/group infrastructure of PublicLab.org itself rather than build a new dashboard. For example, https://publiclab.org/wiki/translation lists some activities, and we could think about recruiting people to follow the Translation /topic/ at https://publiclab.org/tag/translation to get involved. We could make the wiki page the "dashboard" perhaps? I'm just thinking about how to be judicious in creating new systems that would have to be maintained as well.

I have added a section "Reusing existing infrastructure to make a Dashboard-like setup " where I have proposed how we can reuse the existing infrastructure to design a Recruitment Dashboard.

I love that you've mapped out the recruitment flows. It'll be great to think about where the "roadblocks" are and to ensure for example that people who don't have GitHub accounts are supported. You can see a bit of this happening already at https://github.com/publiclab/plots2/issues/9481 but most translators/reviewers may not even be familiar with what GitHub is!

I have added a list of some roadblocks I feel we can work on.

We might have a test page with a variety of HTML content in it, which we attempt to run the translation helper on, and to write a test around this?

I have added a suggestion in "Improving Translation Helper" about a sandbox-type page where all new changes can be seen and tests can be written to compare helper functions.

Please review these changes and share your views about it 😃

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


This is a well-researched proposal @ajitmujumdar25999. Thank you for sharing. I like the breakdown of the Dashboard approach and the re-use of the existing infrastructure.

I wanted to know more about the kind of tests you intend to add...functional, unit, system? I like the sandbox idea, will it need additional maintenance after the project is complete?

Thanks again, and all the best!

Is this a question? Click here to post it to the Questions page.

I wanted to know more about the kind of tests you intend to add...functional, unit, system? I like the sandbox idea, will it need additional maintenance after the project is complete?

I intend to make the sandbox as light as possible it will be like a basic test page to see if UI breaks due to changes in Translation helper,js or HTML, I will add unit and functional tests for all new features and based on requirements try to design it in such a way that it requires minimum maintenance.

Is this a question? Click here to post it to the Questions page.


Reply to this comment...


Login to comment.