Portfolio

Panorama Stitching

The purpose of this project is to stitch two or more images in order to create one seamless panorama image. I will be giving a very brief overview that is able to sufficiently understand the intuition behind how image stitching works.

There are 7 main parts behind how panorama stitching works:

1. 1. Adaptive Non-Maximal Suppression (or ANMS):

      ANMS is used to select a subset of all the key points which have high corner scores and which are averagely spatially distributed in the image. ANMS is easy to implement. Loop through all the initial key points, and for each keypoints compare the corner strength to all other key points.

   2. Feature Descriptor:

      Now that all the corners are found by implementing ANMS, which are also known as feature points, the next step is to describe each of the feature points by a feature vector. This is basically like encoding the information at each feature point by a vector.

   3. Feature Matching:

      Now, you want to match the feature points among the two images you want to stitch together. In computer vision terms, this step is called finding feature correspondences within the 2 images. Pick a point in image 1, and compute the sum of the square difference between all points in image 2. Take the ratio of the best match (lowest distance) to the second best match (second lowest distance) and if this is below some ratio keep the matched pair or reject it. Repeat this for all points in image 1. You will be left with only the confident feature correspondences and these points will be used to estimate the transformation between the 2 images also called Homography.

   4. RANSAC:

      We now have matched all the features correspondences but not all matches will be right. To remove incorrect matches, we will use a robust method called Random Sampling Consensus or RANSAC to compute homography. RANSAC, is an iterative method for estimating a mathematical model from a data set that contains outliers.

   5. Cylindrical Projection: When we are try to stitch a lot of images with translation, a simple projective transformation (homography) will produce substandard results and the images will be strectched/shrunken to a large extent over the edges.

   6. Blending Images: Panorama can be produced by overlaying the pairwise aligned images to create the final output image.

This project is able to stitch any number of given images – maybe 2 or 3 or 4 or 100 and create a panoromic image.

View Project

View Project

Ball-Detection in Nao Robots for RoboCup

Nao robots are star players in RoboCup, an annual autonomous robot soccer competition. UMD is planning to build the Maryland RoboCup team to compete in RoboCup 2022. Using the data collected from the Nao robot’s camera as the training data, where the image names represent the depth of the ball from the Nao robot in centimeters.

In Robocup, the soccer field is bright green, the ball is bright orange, and the goalposts are bright yellow. If we separate the image into these three color classes, we can identify if/where these objects are in the image. A sample pipeline is: 1) the robot acquires an image, 2) classifies each pixel as belonging to one of the color classes (“soccer-field-green”, “ball-orange”, or “goal-post-yellow”), and 3) groups the labeled pixels and classifies the objects. This information is then passed on to a higher-level planning algorithm, which makes high-level decisions like “kick the ball”.

To build the vision pipeline for Nao robots, we will follow a 5 step method:

1. 1. 1. Color Imaging:
               The colors RGB can be represented in a 3D vector space. Think of this as X, Y and Z co-ordinate of a vector space representing colors. A value of [0,0,0] represents pure black, [255,255,255] represents pure white, [255,0,0] represents pure red and so on. Gray is any color with equal values for all the three channels. We can visualize RGB space in three dimensions as a unit cube.

      2. Color Classification:
         Particularly, we are interested in finding the orange pixels because this represents the ball. As mentioned before, in RGB color space each pixel is represented as a vector in R³.

      3. Color Classification using Single Gaussian:
              This is good for most basic cases but is bad for robotics because we said that everything (sensors and actuators) is noisy and we want to model the world in a probabilistic manner. This means that instead of saying a pixel is orange/red we want to say that the pixel is orange with 70% probability and red with 30% probability. Modelling the likelihood as a gaussian is beneficial because a little light variation generally makes the colors spread out in an ellipsoid form, i.e., the actual color is in the middle and color deviates from the center in all directions resembling an ellipse. This is one of the major reasons why a simple gaussian model works so well for color segmentation.

      4. Color Classification using a Gaussian Mixture Model:
         However, if you are trying to find a color in different lighting conditions a simple gaussian model will not suffice because the colors may not be bounded well by an ellipsoid. In this case, one has to come up with a weird-looking fancy function to bind the color which is generally mathematically very difficult and computationally very expensive.

      5. Estimation of distance to the ball:
               Now that we have robustly estimated the pixels which are ‘Orange’, we want to identify the pixels which belong to the orange ball and eventually find the distance to the ball. In the first step, let us identify the pixels which belong to the orange ball. For the second step, one could just fit a simple parametric model (choose a model of your choice) to estimate the distance from different parameters based on the image.

Congrats! You have now built a robust vision system to identify an orange ball, and estimate the distance to it on a nao robot for RoboCup soccer.

View Project

View Project

What is a ‘Good’ Question?

Generating Questions from a Story

Several projects can generate questions given a text such as:
https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00415/107615/PAQ-65-Million-Probably-Asked-Questions-and-What
Now, not all of these questions are going to be great.  Can you write some code that finds the best questions out of the things that get generated?  (As a bonus, if the questions are sorted in some way, that would be better).

This program runs on the list of questions. It identifies good questions and sorts them.

Pre-project:

What makes a question a good question?
A question is considered to be good considering all these categories (Categories devised by me)

• • Relevance and Context
  • Clarity
  • Concision
  • Clarity
  • Open-Ended Nature
  • Depth and Engagement
  • Vagueness

In my model to identify the good-est questions of all, I have decided to incorporate these categories and grade each question through these categories. Some of these categories can be overlapping and contradicting, hence I came up with 4 out of these 7 criteria which can be very crucial in determining the goodness of a question. These 5 pointers are challenging to measure, however, I have created a semi-formal model for each of them considering the text is a story. There were a lot of assumptions that were taken into consideration to come up with these models. I will be briefing about the general model first and will move to each specific criterion model and discuss the assumptions and possibility of extension of that particular model.

Good Question Identifier:

The model will identify questions that are good using a traditional grading system with the help of 2 criteria. a) Relevance and Context, and b) Open Ended Nature. Each criterion is given a grading scale respective to their importance in the formation of a good question and total tallies up to 11 points. The grading scale is described below. The models in each criterion will determine how many points each question is worth. If the combined tally of points from each criterion is more than or equal to 21 points (70%), the question is considered a good question. The program will sort the questions in an order that starts from the good-est question to the worst.

Some of the text-processing ML algorithms I used in my models are:

• • Stopwords filtration
  • Tokenization
  • Stemming and Lemmatizing
  • Matrix Factorization
  • Word sense disambiguation

Here are the 2 criteria and their models:

1. Relevance & Context:
         Stopword filtration is run on the story.txt file along with tokenization which will remove all the punctuations and stop words and tokenize the remaining words storing them in a list. These remaining lists or words are then iterated over and a relative frequency is calculated by the (# occurances/ # total words).Stopwords filtration and tokenization are also run on each question and each word in the question is checked against the list of words from the story.txt and the corresponding relative frequency is calculated. This is how each question will get its respective score and will be graded according to the grading scheme provided below:

   | Relative frequency sum of the question (% of words present from the total words in the text) | Relevance and Context grades |
   | ———– | – |
   | below 0.05 | 1 |
   | 0.05 – 0.10| 3 |
   | 0.10 – 0.20| 4 |
   | 0.20 – 0.30| 5 |
   | 0.30 – 0.40| 6 |
   | 0.40 – 0.50| 7 |
   | above 0.50 | 8 |

2. Open-Ended Nature:
         A question must be open-ended to be considered a good question. To check the open-ended-ness of a question I will be using a model that highly relies on command terms. After a lot of research and development, and the use of a few online resources, I was able to come up with a model on how to test the open-endedness of any question (to some extent). The model goes as follows:

        Check whether the question ends with a “?” or not. If it does, check its first word and test it through the first_word_list to obtain a value of the question. If not, check whether the question contains any command terms and test it through the command_term_list to obtain a value of the question.

        I was able to categorize command terms into 8 different categories according to their open-ended-ness. The model will look up the command term present in the question and the question will be graded according to the rubric. I was also able to sort the first-word list according to the open-endedness of each term. The rubrics are provided below:

   | List of Command terms | Open-Ended Nature grades |
   | ——————— | ———————— |
   | Calculate, Classify, Formulate, Integrate, Measure, Plot, Select | 1 |
   | Define, Determine, Differentiate, Estimate, Find, Label, Recall, State, List, Identify, Predict | 2 |
   | Annotate, Comment, Deduce, Derive, Distinguish, Find, Otherwise, Prove, Solve, Suggest | 3 |
   | Apply, Draw, Describe, Distinguish, Trace, Outline, Use, Verify | 4 |
   | Construct, Demonstrate, Explore, Justify | 5 |
   | Compare, Contrast, Critique, Deduce, Explain, Interpret | 6 |
   | Examine, Investigate, Synthesize | 7 |
   | Analyze, Discuss, Evaluate, Justify, Create, Design | 8 |

Assumptions

The whole model is used with a set of assumptions being:

1. 1. A question will always have exactly one command term present. If multiple are found, the question will be graded only by the first command term found.
   2. If no command terms are found, the question is graded 0 for Open Ended Nature.

Sample Program

Story.txt and Questions.txt are two sample files that I have provided to test the model. Story.txt is a chapter from a short story and Questions.txt is a list of questions regarding the story where each line is a new question.

View Project

View Project

Little Baker Jaaie (Website)

LittleBakerJaaie is a local bakery startup established by a 9-year-old girl from Ahmedabad, Jaaie Shah. She holds multiple records in India’s book of Records and Asia’s book of Records for being the youngest baker. As a freelancing project, I took up a challenge to help Jaaie Shah grow her startup and get recognized at a greater level by creating an E-commerce Bakery Website for the business where people can directly order desserts from the website as well as check out the vast baking catalog curated by Jaaie Shah. I handled the back-end and front-end development to establish and guide website architecture and functionality for a local bakery and also developed and structured a feature-rich client website with a baker and customer interface on WordPress using PHP, HTML, and several plugins, ensuring that all needs were met, and company policies and procedures followed.

Features:

• Site Search
• Footer Navigation
• Product purchase/pre-order button
• Title and SEO description
• FAQ section
• Product features and description
• Product reviews
• Order tracking
•  Product availability filters
• Multilingual support
• Carousels
• Brief information about payment and delivery systems
• Payment and Login auth-0
• Menu
• Breadcrumbs

Visit Website

View Website