World's most advanced storyteller
Our state-of-the-art AI automatically finds stories in your user’s camera roll, then curates them into a photobook, turning photos into memories, events into photobooks.
On our native white-label mobile app, all the AI are done on the edge. That is, the photo analyses are all on-device. No photos leave the phone until they are ordered.
On our webapps, photo analyses are done on our VIQ (Visual Intelligence) server hosted on Amazon. It runs our proprietary storytelling engine and all the related components.
Putting together a Story
Collections SDK takes data from EXIF, meta-descriptors from the Curation SDK analyses and if available, face descriptors plus other phone sensors, then groups relevant photo-sets into a Story.
From “Summer in Berlin”, “Me and Wife in New York last Fall” to “Cats”, “Dogs”, “Meghan and John” and a book with close-ups of Flowers. Your users take photos of the most amazing things in the world. The Collections SDK finds the thread that binds them. Here are three of the most common ones. If none of these describe you, drop us a line and let’s talk.
Time and Location-based Collections
These are our default in our white-label mobile apps as they are the fastest to generate since we only require time-stamps and location information typically found in the EXIF tag of photos taken on a mobile device. Using proprietary behavior-predictive algorithms, we group photos using a time and location proximities.
The algorithm first determines where your “home” base city is, and your typical travel patterns to determine how far you go before its “a trip”.
For someone who lives in San Francisco and commutes to San Jose (120km) every day for work compared to someone in tiny Singapore where the typical commute is 10 km, the definition of a “trip” is subjective and relative. Our algorithm figures it all out. Even if the user moved from compact Ho Chi Minh to work in sprawling Los Angeles, the algorithm will automatically re-calibrate over time.
What about privacy?
Because the entire SDK works natively on the device, nothing gets stored nor transmitted to anyone. Your app won’t know where the user’s “home” is, our app won’t know it either. We calculate a locus of numbers which describes a “zone”.
Person-based Collections
This feature requires face recognition and person-clustering information to be provided to the Collections engine. This is supported by our iFACE SDK, but it can also work with your own Face Group information via APIs. If you have a manually curated album (if you are a wedding or school yearbook photographer), then we can bypass the Collections engine totally in our webapp.
By analyzing the Frequency and Recency of the occurrences of two or more persons in the same photos and at events, we are able to predict their social relationships and “social-distance”, then group them into photo albums naturally to optimize conversion.
Theme-based Collections
These are Collections based on specific themes powered by the AI Image Recognition models. It could be wedding photos, night shots, an anthology of Flowers, Birds, Sports, or any other recurring themes and patterns that we sense in the phone gallery. Get on our newsletter to stay abreast of our developments in this area!
(This is currently in Beta and not yet available to evaluate.)
Curation SDK = Eyes + Brains + Storyteller
The Curation engine uses advanced Computer Vision, Face and AI algorithms to annotate each photo by analyzing it for various metrics.
Picture Quality
First, each photo is scored for various attributes like its exposure, brightness, focus, contrast, and other color histograms. This helps remove blur shots, shots of the floor, hands covering the lens type shots.
We know. We have all been there.
Similar vs Duplicates
We studied how a person would perceive how similar photos are to each other, we measure multiple levels of similarity using multi-scalar algorithms to distinguish between photos that “kinda look the same at a glance” vs “duplicates”. We know consumers love snapping multiple shots each time. No problemo. There is even a “fussiness” parameter you can set to tune the engine to the level of similarity you would like to see in your application. This means we are able to guarantee repeat shots are eliminated, and pages has visual variety when you are browsing quickly through the photobook.
Theme-based Collections
Many people use their phone camera as a sticky note reminder and a repository of snippets. See a restaurant review in the in-flight magazine? Snap. IKEA furniture pickup-tag? Snap. Subway map of Tokyo? Snap.
We have trained a sizable (and growing) cess-pool of over 100 different types of “the-crap-we-shoot”; stuff like whiteboards, For-Sale signs to driver’s licenses and almost any other receipts/price-tags/maps you can think of. So we proudly call this AI model our De-crapperizer. It sniffs out all the crap and eliminates them from your album.
Food Porn or Latte Art?
This is where the ‘gram has influenced our product. People love to take photos of food and latte foam art. You know…that heart shape, teddy bear made with steamed milk and espresso? We recognize these, and we do magical things with it. If you have ever had a vacation in a culinary wonderland like Paris, or Vegas, I am sure you are guilty of that too! It does not have to be a Robuchon dinner. It might just be Cheesecake factory…pretty pastries complete our vacation stories. But a cake a page is pushing it, don’t you think?
Scene Intelligence
Being able to distinguish between a Landscape shot vs a photo of People, or an Indoor Stage Performance helps us craft the book better and make it more human. A photo book is a story waiting to be told, to be cherished, and to be immortalized. This is used in various different ways in our white-label mobile app and our online photobook designer webapp.
Safe-Crop-Zone: Don’t Crop the Face!
Life is not perfect. To make an interesting page collage, you need to crop photos. But how do you intelligently make sure you do not crop out the salient parts of the photo? Most importantly, people really do not like the faces of their little Johnny boy being cropped away. So the Curation SDK will also return a “safe-crop-zone” of each and every photo we select for the book, so you know where the no-go zones. Ignore these hints at your own risk!
Elastic Storytelling
In every Story, there are micro-stories. From a 15-day Mediterranean cruise to Ethan’s 7th birthday party, every moment has its special place in our minds and hearts. The Elastic Storyteller algorithm resolves mini-groups of photos to fit any number of pages, making sure micro-stories stay within a spread. Our mobile app can automatically take the same story and make a 24 page book, or help you increase revenue by generating a book using more pages…but always retaining the story integrity across the book.
Layout Engine - the Book Builder
This component takes the page-grouping and safe-crop information from the Curation SDK and builds a book that is ready for print. It comes with its own brains to ensure faces are never cut off regardless of what type of layout options a user may choose. It then prepares the pages to be rendered in print resolution, which can be done both natively, or in the cloud just before book purchase.
Special Collages – Food Porn and Cafe-art
So we know you have a bunch of food and latte art photos. What do we do with that?
Photos of food from any vacation is so much a part of the experience of any city. But you don’t really want your food pics on so many pages.
Our layout engine is tuned to automatically cluster them and create periodic food collage pages within the book.
Wide-Screen Panorama
Algorithmic Collages
In modern mobile photography, photos can be 4:3, 16:9, 1:1 and even 2:1 aspect ratios. It’s no longer a simple case of portrait or landscape!
We crafted algorithms that are inspired by the beauty of nature and aesthetic design ratios. This means that there are no heavy templates and clip-art to download. Every layout option is an optimization algorithm that works across both facing pages, rather than a fixed alpha-channel PNG which are restrictive. The algorithm ensures layout options of opposite pages in a spread are coherent, aligned and simply, exquisite!
Captions - Storytelling
Page layouts have two modes: with or sans Captions. It then automatically adjusts vertically to make space for page Captions.
Face Intelligence
The face is the most emotionally engaging part of any photo and story. Hence we devoted a significant part of our AI in making better-informed decisions on any faces detected in the photos for any product. Whether this is a collage canvas of 20 pictures, or a photobook, our AI naturally considers faces.
Face Quality
Assigns a score of 0 to 100 on each face in a photo Face Score is based on the following components:
- Face Frontal (measures how much the face is looking at the cameraman).
- Eyes Open (measures if eyes are open).
- Smile Score (measures the amount of smile).
- Face Size (for multi-face photos, assigns weight accordingly).
- Runs totally locally on Mobile (Android, iOS), Desktop (Windows, Mac OSX) and also available as a Service online.
Face Clustering
- Clusters like-faces into buckets.
- Tested up to ~38,000 different unique persons from a total of 300,000 files.
- Based on our proprietary Face Recognition algorithm which uses a Convolutional Neural Network to derive 128 facial metrics to describe the unique characteristic of each face.
- Runs totally locally on Mobile (Android, iOS), Desktop (Windows, Mac OSX) and also available as a Service online.
Face Recognition
- Based on our proprietary Face Recognition algorithm which uses a Convolutional Neural Network to derive face metrics from over 60 different face landmarks.
- Face metrics are derived from faces regardless of orientation (Pitch, Yaw, Rotation with respect to camera).
- Robust to partial obfuscation of face (upto ~20%).
- Runs totally locally on Mobile (Android, iOS), Desktop (Windows, Mac OSX) and also available as a Service online.
- Works even in low light, partially shadowed.
- Works even with glasses on/off, head-dress, covered forehead, partially covered chin (scarves etc).
- Runs totally locally on Mobile (Android, iOS), Desktop (Windows, Mac OSX) and also available as a Service online.
photobook.ai uses AI to automatically create photobooks and other printable products. We provide white-labelled app-to-print native apps and progressive web-apps. Our white-label platforms run globally in over 20 markets, doing thousands of orders each day.
© 2024 photobook.ai