Building a face recognition API with face_recognition, Flask and PostgreSQL
A couple of months ago I read about PimEyes a service that allows for finding images across the web that contain a matching face. There’s a little controversy behind the service because it doesn’t vet its users strictly which means that the system can be abused.
This got me thinking about how such a technology was built, as the face recognition libraries I had seen in the past required loading in images to conduct a comparison and at the scale that Pimeyes would be operating on this would be really inefficient.
In order to return results across multiple faces quickly there would need to be some alternative means of carrying out this face comparison on the database layer.
After a bit of searching I found an excellent, but broken example using PostgreSQL’s
that saves two vectors of the face’s descriptors and uses euclidean distance to return records that are above a certain threshold.
After fixing the incorrect SQL statement I was able to save an image’s descriptors and then use another image’s to
return the original image as a match.
How it works
The Python code makes use of a couple of libraries:
opencv— Used to read the image to be processed
dlib— Used to load a Histogram of Oriented Gradients (HOG) face detector used to find faces in the uploaded images, returns a list of coordinates in the image for a bounding rectangle of the face
face_recognition— Used to get a list of face descriptor encodings, a 128-dimensional array of the points for the faces landmarks
Just a heads up — dlib can be a pain to install, especially on a Mac. The face_recognition repo has a good set of instructions of installing the library https://github.com/ageitgey/face_recognition
The PostgreSQL database saves the 128-dimension array as two 64-dimension
cubes as Postgres supports 64-dimension
out of the box (there are ways to increase this size if needed).
When querying the database to find matching faces the database compares the Euclidean distance between the two input
cubes and the
cubes in the records with a smaller Euclidean distance, meaning a face that is more
similar to the face used as input.
Creating a basic application
The example I based my work off came with two scripts, one for adding a face to the database and another for querying the database for matching faces in the database. These scripts worked well but were limited to local images, in order to build something a little more useful I wrapped those scripts in a Flask API, allowing for users to POST images to be added and searched with.
For the API to work the database needs to be set up with the cube extension enabled and the tables created.
When adding a face I wanted to return the ID that the face was saved under in the database so systems that use the API
have a means of linking image records to the records in the system (this linking is already handled in the database via
When searching for a face the API returns the
image_id values stored against the matching faces so
consuming systems can return those as part of their response.
There’s a few ways the face recognition processes could be built into more advanced applications. The following sprung to mind as I was working on it:
- Integrating into a CMS such as Django by building an extension that uses the Python code to find faces in images uploaded by users and storing these as a separate model which links the image to the faces found and can be used in queries or to build a graph of images in order to explore the relationships between those in the images
- Integrating into other CMSs in a similar matter by calling the API to add and find faces, although it might be easier to use a face recognition library in the language the CMS is written in to reduce execution time (I started looking at doing this with face-api.js and Strapi for NodeJS)
- Building a specialised web crawler to create a face search for websites that can’t rely on tagged data such as event photography websites so that users could use a webcam capture to find images of themselves across all the images taken
I was quite happy with the discoveries I’d made building this basic API wrapper. I learned about a few new concepts like
cube type and how face recognition works.
I’m currently playing around with Strapi, seeing how I could extend it’s functionality to do create a graph of faces in different images so these can be queried via GraphQL as I think this will be an interesting application of the technology.