Note: This is a draft and some sections are under development
About me
Name: Chinmay Pandhare
Email: ccpandhare@gmail.com
GitHub Profile: https://github.com/ccpandhare
Gitter Nick: ccpandhare
Institute: BITS Pilani (Pilani Campus)
Degree: B.E. Electrical and Electronics Engineering (Class of 2020)
Location: Pilani, Rajasthan, India (GMT + 5:30)
Developing Image Sequencer as a Library
Image Sequencer is a non-destructive image processing library built with JavaScript which creates a new image at each step in a sequence of steps.
Current State:
Image Sequencer is in early-development phase. The core functionality (addition of a step, removal of a step, insertion of a step, etc.) is being implemented. There are a few modules present. Calculations aren't GPU accelerated. As a result, there is a lag when images larger than 700KB are handled.
Aim of my project:
__
The aim of my Project is to expand the module library by implementing powerful and useful image processing modules (As listed at the end of this proposal) with possible integration with GPU.js for enabling GPU-backed calculations and hence enhance performance, reliability and speed. And, ultimately, to develop Image Sequencer as an importable library for image manipulation. Also, support will be developed for JSON input. This is explained below in detail.
Current functioning of Image Sequencer
This chart shows how Image Sequencer currently works. There are certain issues attached to the current implementation of Image Sequencer:
- Long call syntax --- This can be improved upon.
- Only one image is handled per instance of Image Sequencer
- Direct DOM manipulation --- This makes Image Sequencer demo specific. i.e, Dependant on the HTML UI
- No JSON support --- This makes integration of Image Sequencer into large-scale projects impractical.
- Time Lag --- When a step is added, sequencer.run() runs through all the steps --- this creates an unnecessary time lag.
- The outputs of different steps aren't stored and hence can't be accessed at will. The library simply injects the output images into the UI.
- No functionality for modules to send logs or non-image data to user.
- Non GPU-accelerated --- This slows down speed for images > 700KB
My project aims at restructuring the code and expanding the module base which will lead to faster and neater usage.
Post-Project functioning (Draft)
This chart shows how I plan to make Image Sequencer work. These would be my milestones:
- Completing the core functionality --- addStep/s, removeStep/s, insertStep/s, getOutput, stepLog.
- Supporting JSON type input & output to enable easy use and shorter call syntaxes.
- Enhancing speed with the help of GPU accelerated calculations using GPU.js.
- Accepting logs / non-image outputs from modules.
- Enabling multiple image handling
- Reducing dependancy on UI and DOM Manipulation
- Implementing modules (non-exclusive list at end of the proposal)
- Publish to npm.
Post-Project Usage Examples (Both methods)
Method - 1 (Similar to the existing model):
Method - 2 (JSON Model):
Timeline (Coding Days: May 30 to Aug21)
Week 1 & 2 (May 30 to June 11)---Core Functionality and JSON
Implementing the core functions addStep/s, removeStep/s, insertStep/s, getOutput, stepLog, complete with JSON input/output support. This will include most of the restructuring work. ImageSelect.js will no longer have a 'special module' status as it currently does.
Week 3 (June 12 to June 18)---Multiple Image Support and GPU.js integration
Support will be added for handling multiple images. User would be able to access data relating to any step on any image at will.
GPU.js will be integrated with Image Sequencer. So that the matrix calculations of modules are GPU accelerated, whenever possible. This will include changes in /src/PixelManipulation.js and other files.
Community review for Core Functionality and JSON.
Week 4 (June 19 to June 25)---Module Logs
Module logs will be returned to user. A method getOutput will be introduced. This will enable modules to send non-image data to the user. For example, A QR Code reading module can send detected data to the user.
Community review for Multiple Image Support and GPU.js integration.
Weeks 5 & 6 (June 26 to July 9)---Module Implementation
Modules in the list of module candidates will be implemented along with Additional modules as suggested by the community. Also a special module, 'Color Space' will be implemented which changes the colour space of the image to user defined functions of r, g, b, a.
Community review for Module Logs
Weeks 7 & 8 (July 10 to July 23)---Work on Demo
Demo will be reworked, with a better UserInterface.js. UserInterface.js will be made demo-independent --- It's function will be independent of the implementation where Image Sequencer is being used.
Community review for Module Implementation.
Weeks 9 to 13 (July 24 to August 21)---Final Work, Fixes, Suggestions
This period will be for bug-fixing, goal-expansion, and implementing community suggestions. Also, My college starts functioning from July 31. So my contribution might decrease a bit in this period. This will officially conclude my work for Image Sequencer, though I'll still be associated with the project and contribute to it, as and when possible.
Contributions
This is my fork of the Image Sequencer project: https://github.com/ccpandhare/image-sequencer
These are a few of my contributions to publiclab/image-sequencer:
- Added removeStep functionality (Built) [PR #17]
- Fixed addStep functionality (Merged) [PR #12] [Issue #2]
- Added Crop Module (Merged) [PR #8] [Issue #5]
- Bugfixes in ImageThreshold.js and ImageSelect.js (Merged) [PR #6] [Issue #2]
- Detected and reported issues #19, #15, #14, #13, #3, #2
Experience
I am a Web Developer and a Python enthusiast. I have worked on many technologies till date and have done that over the past six years.
Some of my notable projects:
- Ziolk : A tool for designing T-Shirts online (JavaScript - Canvas - Fabric.js)
Basic Functionality : Adding shapes, text, images, background removal form imported images, flipping objects along X and Y axes.
GitHub Link : https://github.com/ccpandhare/ziolk - Nalanda Lite : A script to auto-download lecture slides (Python - BeautifulSoup)
Basic Functionality : Download/Update Lecture slides from our college portal, Nalanda and store them on the user's laptop.
GitHub Link : https://github.com/ccpandhare/nalandalite - Document management portal for the District Court of Jhunjhunu, Rajasthan State, India in association with the Media Lab, BITS Pilani. (PHP - mySQL)
Skill Set
Front-end Languages : HTML, CSS, JavaScript
Front-end development frameworks used : SASS, HAML
Front-end deployment frameworks used : jQuery, Angular.js, D3.js, Bootstrap, Materialize, Fabric.js.
Back-end Experience : PHP, mySQL, Node.js
Teamwork
I am a part of DVM BITS Pilani. We are a group of students who create Android Applications, Websites and Videos for the various tests of our college. This is some of the latest work undertaken by DVM:
http://bits-oasis.org , http://bits-bosm.org , http://bits-apogee.org
List of Module Candidates (non-exhaustive)
This list is compiled from the original Readme.md of Image Sequencer and my own online searches.
__
- rExtract https://github.com/linuxenko/rextract.js
- Histogram https://www.npmjs.com/package/histogram
- Flood Fill https://github.com/hughsk/flood-fill
- Blink Diff https://www.npmjs.com/package/blink-diff
- Pixel Match https://www.npmjs.com/package/@schornio/pixelmatch
- PNGJS Image https://github.com/yahoo/pngjs-image
- CannyJS https://github.com/yuta1984/CannyJS
- JavaScript Edge Detection http://codepen.io/taylorcoffelt/pen/EsCcr
- JSQRCode https://github.com/vicapow/jsqrcode
- quaggaJS https://serratus.github.io/quaggaJS/
- jquery.facedetection https://github.com/jaysalvat/jquery.facedetection
- Popular Image Filters
13 Comments
@warren @liz Please review!
Reply to this comment...
Log in to comment
Sorry, there was some formatting error earlier. I have fixed that now. Was working fine when I pasted it, but got distorted upon submission.
Reply to this comment...
Log in to comment
Hi, this is very thorough and impressive, thank you!
Just some notes and questions, in no particular order:
For GPU, I think we should make this optional, if possible, and maybe we need to think about where swapping in a GPU functionality should occur? Inside each module, I guess? Some uses may not have a GPU - for example on server side use.
Likewise, JSON should be optional -- perhaps JSON could be passed into the constructor method of ImageSequencer. But I think simple function-based calls are very important as people may want to simply interface with a UI, or with another program using them, without having to assemble a more complex JSON object to use the system. For example, the demo uses a DOM click event to add a step.
For output data, I think there's a balance to be struck on whether modules should be stateful -- like, do they store their last output? Or is that handled at the level of the sequencer, so that modules can remain very simple? I think I am leaning towards the latter, but tell me what you're thinking.
Haha, ok - your examples actually address most of my questions. Great proposal!
I'd also like to think more about testing. How can we run tests on this? The current attempt to do nodejs based testing is having some trouble, but we might come across an easy solution with time. An alternative is to use Jasmine and PhantomJS, instead of node, not for the DOM, but for the standard browser-based Image and Canvas APIs, as we do in https://github.com/publiclab/PublicLab.Editor/
can you talk more about this?
Thank you!
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
@warren
Regarding the GPU, yes. Some users may not have a GPU. We'll have to use a JS fallback for then. That shouldn't be a problem, as, the module which I am thinking of implementing, GPU.js, does that by itself.
Not really. Modules like GreenChannel, NdviRed won't need one as the matrix calculations are done in PixelManipulation.js. So, implementing GPU-backed calculations in PixelManipulation.js would suffice for quite a few modules. And for the rest, as you said, integration will have to be done in every module.
Yes. That's what I was trying to say. I guess my explanation isn't good enough. Maybe I should rephrase parts of my proposal a bit.
As far as statefulness is concerned : I know -- the idea of 'Image Sequencer' does make us intuitively think of a stateless model. Like the current one. But, that is something which works for the demo, because we inject a DIV in the DOM when we call
draw(). This is something which wouldn't work for other implementations. The aim of Image Sequencer is to enable the user to make comparisons and document different steps. So if we are able to only provide the last step, it would take away Image Sequencer's USP. So we have two options : 1) Store the output for every step. 2) Run the sequencer whenever we want an intermediate output. The second option has a greater time complexity. So In my opinion, storing the output after every step is a good solution. What do you think about this?Yes, I am working on that part now. I am trying out the node.js based implementation and fixing it up. I'm sure we'll be able to fix that issue. Testing shall be an important part, undoubtedly.
Image Sequencer promises to run a sequence over one or more images. Currently, for one input image, one instance of Image Sequencer is needed. I feel this could be better implemented by storing all the images in one module itself. Using the same
steps[]array and same functions.Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
@warren so should I make any other changes?
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
Hi, perhaps you could present a rough schedule and ordering of how you'd approach the projects? Thank you!
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
@warren I've added a rough schedule. Does that look fine?
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
I'll also add the tests and node support parts into the proposal
Reply to this comment...
Log in to comment
Thanks, the schedule looks great. I really appreciate your thoroughness here. I hope to look a bit at your recent work on the repository as well in the next few days, thanks for your patience as I'm traveling for work :-)
Reply to this comment...
Log in to comment
Okay! Although, this draft is a few edits behind the current one. I've submittted an official draft via the SummerOfCode website. I'll update this one too.
Reply to this comment...
Log in to comment
I'm having issues editing it here. The editor isn't loading at all.
Reply to this comment...
Log in to comment
Hmm, can you file a bug? If it doesn't work, no worries, I'll check both places, but let's figure this out!
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
I had a look. The editor did open, but took a lot of time to open. Maybe it was a connectivity issue. I'll update this soon.
Reply to this comment...
Log in to comment
Login to comment.