opencv - finding similar images from a database using image only not via tag -

well question simple want find similar images given query image, similar tineye does. suppose have shirt following description

sleeve length : full

collar : present

pattern : striped

(the above data give feel of image dont have data)

first image query image , next should output of similarity finding algorithm. based on example have flexibility can show user image changed color, can see images have same pattern, same collar type or sleeve length. have show output visually similar.

there similar thread on stack link stack , not there many other. confused approach follow.

in case dont have search in category have search in same category if input shirt search in shirt category only. part has been done.

so question approaches handle problem. color no big issue. color information can extracted through color histogram. lets input tshirt round neck i.e. without collar, half sleeve , printed @ center text. output should images similar half sleeve, round collar, , printed text @ center. thought text may vary. tried k-means clustering , p-hash didnt work. please enlighten me

ps : have find similar images not duplicates.

i try split problem 3 smaller problems:

• checking whether image shows shirt long or short sleevs
• checking pattern (stipped, plain, else?)
• determining color of shirt

---

checking whether image shows shirt long or short sleevs
1 in opinion easiest. mentioned have category name, basing on google graphics seems may not obvious whether shirt or tshirt has long or short sleevs.
solution quite simple:

• find face on image
• use grabcut algorithm extract face mask image
• mask face (so after step face left - everythin else black). note step not necessary - i've mentioned only, because it's shown on final image.
• convert image hsv color space
• using face mask calculate histogram h , s color channels of face (without rest of image)
• calculate projection of hsv image using histogram previous step. regions color (in hsv) similar color of face - regions contains skin.
• threshold result (there noise :) )

the final result of algorithm black , white image shows skin regions. using image can calculate number of pixels skin , check whether skin on face or maybe somewhere else. can try find contours - both solututions give chance check if hands visible. yes - shirt has short sleevs, no - long sleevs.

here results (from top-left corner - original image, face mask (result of grabcut algorithm), masked face, hsv image, result of calculating projection, result of using threshold on previous one):
can see, unfortunetely fails image 3, because face in similar color shirt pattern (and face color quite close white - wrong guy, should spend more time outside ;) ).

source quite simple, if don't understand feel free ask:

import cv2 import numpy np   def process_image(img, face_pos, title):     if len(face_pos) == 0:         print 'no face found!'         return     mask = np.zeros((img.shape[0], img.shape[1]), dtype=np.uint8) #create mask same size image, 1 channel. mask initialized zeros     cv2.grabcut(img, mask, tuple(face_pos[0]), np.zeros((1,65), dtype=np.float64), np.zeros((1,65), dtype=np.float64), 1, cv2.gc_init_with_rect) #use grabcut algorithm find mask of face. see grabcut description more details (it's quite complicated algorithm)     mask = np.where((mask==1) + (mask==3), 255, 0).astype('uint8') #set pixels == 1 or == 3 255, other pixels set 0     img_masked = cv2.bitwise_and(img, img, mask=mask) #create masked image - show result of grabcut     #show images     cv2.imshow(title, mask)      cv2.imshow(title+' masked', img_masked)      img_hsv = cv2.cvtcolor(img, cv2.color_bgr2hsv) #convert image hsv     channels = [0,1]     channels_ranges = [180, 256]     channels_values = [0, 180, 0, 256]     histogram = cv2.calchist([img_hsv], channels, mask, channels_ranges, channels_values) #calculate histogram of h , s channels     histogram = cv2.normalize(histogram, none, 0, 255, cv2.norm_minmax) #normalize histogram      dst = cv2.calcbackproject([img_hsv], channels, histogram, channels_values, 1) # calculate project (find pixels color similar color of face)     cv2.imshow(title + ' calcbackproject raw result', dst)      ret, thresholded = cv2.threshold(dst, 25, 255, cv2.thresh_binary) #threshold result of previous step (remove noise etc)     cv2.imshow(title + ' thresholded', thresholded)      cv2.waitkey(5000)     #put partial results 1 final image     row1 = np.hstack((img, cv2.cvtcolor(mask, cv2.color_gray2bgr), img_masked))     row2 = np.hstack((img_hsv, cv2.cvtcolor(dst, cv2.color_gray2bgr), cv2.cvtcolor(thresholded, cv2.color_gray2bgr)))     return np.vstack((row1, row2))   paths = ['1.jpg', '2.jpg', '3.jpg', '4.jpg'] haar_cascade = cv2.cascadeclassifier('c:\\devtools\\src\\opencv\\data\\haarcascades\\haarcascade_frontalface_default.xml') #change path face cascade - it's inside opencv folder  path in paths:     img = cv2.imread(path)     face_pos = haar_cascade.detectmultiscale(img, 1.3, 5, cv2.cascade_find_biggest_object)     if len(face_pos) == 0: #if haar cascade failed find face, try again different (more accurate, slower) settings         face_pos = haar_cascade.detectmultiscale(img, 1.1, 3, cv2.cascade_find_biggest_object)     result = process_image(img, face_pos, path)     cv2.imwrite('result_' + path, result) #save result 

---

checking pattern (stipped, plain, else?) , determining color of shirt
here try extract (mask) shirt image , operate on it. achieve try use similar approach in previous part - grabcut algorithm. initializing might harder time. quite easy (but not perfect) solution comes mind is:

• set rect around whole area (leave few pixels on each side)
• initialize mask sure foreground value in middle of image - draw circle in middle using sure foregroung "color"
• set mask sure background in corners of image
• set mask sure background in face rectangle (the 1 founded using haar cascade in step "checking whether image shows shirt long or short sleevs")

alternatively can initialize whole mask sure foreground or possible foreground , use watershed algorithm find big white area (which backgrund). once have area - use background.
using 2 solutions give best results.

you can try easier solution well. looks images has got shirt not background, skin or else sligtly on center of it. here: can analyze part of shirt. can try localize sure shirt part of image using haar cascade - find face , move founded rectangle down.

once have masked shirt can calculate parameters. 2 things try are:

• convert hsv color space , calculate histograms (2 separates - not 1 did in previous step) hue , saturations channels. comparing histograms 2 shirts should give chance find shirt similar colors. comparing histogram use (normalized) correlation coefficients.
• use fourier transform see frequencies common in shirt. plain shirts should smaller frequencies stripped.

i know solutons aren't perfect, hope helps. if have problems or questions - feel free ask.

---

//edit:
i've done simple pattern comparision using fourier transform. results are... not good, not bad - better nothing, not perfect ;) it's point start.
package code , images (yours + google) here. code:

import cv2 import numpy np collections import ordereddict import operator   def shirt_fft(img, face_pos, title):     shirt_rect_pos = face_pos[0]     # print shirt_rect_pos     shirt_rect_pos[1] += 2*shirt_rect_pos[3] #move down (by 2 * height) rectangle face - point shirt sample     shirt_sample = img[shirt_rect_pos[1]:shirt_rect_pos[1]+shirt_rect_pos[3], shirt_rect_pos[0]:shirt_rect_pos[0]+shirt_rect_pos[2]].copy() #crop shirt sample image     shirt_sample = cv2.resize(shirt_sample, dsize=(256, 256)) #resize sample (256,256)     # cv2.imshow(title+' shirt sample', shirt_sample)      shirt_sample_gray = cv2.cvtcolor(shirt_sample, cv2.color_bgr2gray) #convert gray colorspace      f = np.fft.fft2(shirt_sample_gray) #calculate fft     fshift = np.fft.fftshift(f) #shift - brightest poitn in middle     # fshift = fshift.astype(np.float32)     magnitude_spectrum = 20*np.log(np.abs(fshift)) # calculate magnitude spectrum (it's easier show)     print magnitude_spectrum.max(), magnitude_spectrum.min(), magnitude_spectrum.mean(), magnitude_spectrum.dtype     magnitude_spectrum = cv2.normalize(magnitude_spectrum, alpha=255.0, norm_type=cv2.norm_minmax, dtype=cv2.cv_8uc1) #normalize result , convert 8uc1 (1 channe 8 bits - unsigned char) datatype     print magnitude_spectrum.max(), magnitude_spectrum.min(), magnitude_spectrum.mean(), magnitude_spectrum.dtype     # cv2.imshow(title+' fft magnitude', magnitude_spectrum)     magnitude_spectrum_original = magnitude_spectrum.copy()     # temp, magnitude_spectrum = cv2.threshold(magnitude_spectrum, magnitude_spectrum.max()*0.75, 255.0, cv2.thresh_tozero)     # temp, magnitude_spectrum = cv2.threshold(magnitude_spectrum, 125, 255.0, cv2.thresh_tozero)     # temp, magnitude_spectrum = cv2.threshold(magnitude_spectrum, 250, 255.0, cv2.thresh_tozero_inv) #clear brightest part     temp, magnitude_spectrum = cv2.threshold(magnitude_spectrum, 200, 255.0, cv2.thresh_tozero) #clear values 0 200 - removes noise etc     # cv2.imshow(title+' fft magnitude thresholded', magnitude_spectrum)     # cv2.waitkey(1)      # if chr(cv2.waitkey(5000)) == 'q':         # quit()      # return fshift     return shirt_sample_gray, magnitude_spectrum_original, magnitude_spectrum  paths = ['1.jpg', '2.jpg', '3.jpg', '4.jpg', 'plain1.jpg', 'plain2.jpg', 'plain3.jpg', 'plain4.jpg', 'stripes1.jpg', 'stripes2.jpg'] haar_cascade = cv2.cascadeclassifier('c:\\devtools\\src\\opencv\\data\\haarcascades\\haarcascade_frontalface_default.xml') #change path face cascade - it's inside opencv folder  fft_dict = ordereddict() results_img = none  path in paths:     img = cv2.imread(path)     face_pos = haar_cascade.detectmultiscale(img, 1.3, 5, cv2.cascade_find_biggest_object)     if len(face_pos) == 0: #if haar cascade failed find face, try again different (more accurate, slower) settings         face_pos = haar_cascade.detectmultiscale(img, 1.1, 3, cv2.cascade_find_biggest_object)     # result = process_image(img, face_pos, path)     # cv2.imwrite('result_' + path, result) #save result     results = shirt_fft(img, face_pos, path)     if results_img none:         results_img = np.hstack(results)     else:         results_img = np.vstack((results_img, np.hstack(results)))     fft_dict[path] = results[2]  similarity_dict = {} cv2.imshow('results_img', results_img) cv2.waitkey(1)   #for each image calcualte value of correlation each other image in range(len(fft_dict.keys())):     j in range(i+1, len(fft_dict.keys())):     # j in range(i, len(fft_dict.keys())):         key1, key2 = fft_dict.keys()[i], fft_dict.keys()[j]         print 'pair: ', key1, key2          img1 = fft_dict[key1]         img2 = fft_dict[key2].copy()         # img2 = img2[10:246, 10:246]         correlation = cv2.matchtemplate(img1, img2, cv2.tm_ccorr_normed)         # correlation = cv2.matchtemplate(img1, img2, cv2.tm_sqdiff_normed)         # print correlation         print correlation.shape, correlation.dtype, correlation.max()         similarity_dict[key1 + ' - ' + key2] = correlation.max()         # similarity_dict[key1 + ' - ' + key2] = correlation  #sort values (from best worst matches) sorted_similarity_dict = sorted(similarity_dict.items(), key=operator.itemgetter(1), reverse=true) print "final result: " in sorted_similarity_dict:     print   cv2.waitkey(50000) 

some lines commented - can try use them, maybe achieve better results.
basic algorithm quite simple - each image:

• cut shirt sample image (just move down rectangle face 2* height)
• convert rect gray colorspace , resize (256, 256)
• calculate fft of sample
• calculate magnite spectrum of fft transform
• normalize (from 0 255)
• threshold (clear values <200) - remove noise etc.

now can calculate normalized cross corelation of image between shirt samples. high result -> similar samples. final results:

('plain1.jpg - plain3.jpg', 1.0)   ('plain3.jpg - plain4.jpg', 1.0)   ('plain1.jpg - plain4.jpg', 1.0)   ('stripes1.jpg - stripes2.jpg', 0.54650664)   ('1.jpg - 3.jpg', 0.52512592)   ('plain1.jpg - stripes1.jpg', 0.45395589)   ('plain3.jpg - stripes1.jpg', 0.45395589)   ('plain4.jpg - stripes1.jpg', 0.45395589)   ('plain1.jpg - plain2.jpg', 0.39764369)   ('plain2.jpg - plain4.jpg', 0.39764369)   ('plain2.jpg - plain3.jpg', 0.39764369)   ('2.jpg - stripes1.jpg', 0.36927304)   ('2.jpg - plain3.jpg', 0.35678366)   ('2.jpg - plain4.jpg', 0.35678366)   ('2.jpg - plain1.jpg', 0.35678366)   ('1.jpg - plain1.jpg', 0.28958824)   ('1.jpg - plain3.jpg', 0.28958824)   ('1.jpg - plain4.jpg', 0.28958824)   ('2.jpg - 3.jpg', 0.27775836)   ('4.jpg - plain3.jpg', 0.2560707)   ('4.jpg - plain1.jpg', 0.2560707)   ('4.jpg - plain4.jpg', 0.2560707)   ('3.jpg - stripes1.jpg', 0.25498456)   ('4.jpg - plain2.jpg', 0.24522379)   ('1.jpg - 2.jpg', 0.2445447)   ('plain4.jpg - stripes2.jpg', 0.24032137)   ('plain3.jpg - stripes2.jpg', 0.24032137)   ('plain1.jpg - stripes2.jpg', 0.24032137)   ('3.jpg - stripes2.jpg', 0.23217434)   ('plain2.jpg - stripes2.jpg', 0.22518013)   ('2.jpg - stripes2.jpg', 0.19549081)   ('plain2.jpg - stripes1.jpg', 0.1805127)   ('3.jpg - plain4.jpg', 0.14908621)   ('3.jpg - plain1.jpg', 0.14908621)   ('3.jpg - plain3.jpg', 0.14908621)   ('4.jpg - stripes2.jpg', 0.14738286)   ('2.jpg - plain2.jpg', 0.14187276)   ('3.jpg - 4.jpg', 0.13638313)   ('1.jpg - stripes1.jpg', 0.13146029)   ('4.jpg - stripes1.jpg', 0.11624481)   ('1.jpg - plain2.jpg', 0.11515292)   ('2.jpg - 4.jpg', 0.091361843)   ('1.jpg - 4.jpg', 0.074155055)   ('1.jpg - stripes2.jpg', 0.069594234)   ('3.jpg - plain2.jpg', 0.059283193)   

image shirt samples, magnitude spectrums (before , after threshold) here:

the images names (in same order samples on big image): ['1.jpg', '2.jpg', '3.jpg', '4.jpg', 'plain1.jpg', 'plain2.jpg', 'plain3.jpg', 'plain4.jpg', 'stripes1.jpg', 'stripes2.jpg'] can see, thresholded images quite similar samples same pattern. think solution work better if find better way compare images (thresholded magnitude spectrums).

edit2:
simple idea - after crop shirt samples lot of shirts, can try train classifier , recognize patterns them using classifier. tutorials training haar or lbp(local binary pattern) cascades.