VueStudio: Behind The Scenes

Reading Time: 4 minutes

Vue.ai’s Shoppable Video solution – VueStudio, is an advanced shoppable video technology solution for marketers and merchandisers, who want to stay ahead of the curve with visual stories that sell instantly. Using Vue.ai’s Image Recognition and Computer Vision technology, VueStudio can make every video shoppable, providing the flexibility to edit, create and design each video based on specific brand requirements. VueStudio lets retailers curate collections and create themed landing pages with video, those that can be syndicated across other channels as well.  

But, how does the magic REALLY happen?

Taking a few steps back, here is a peak into  what really goes on behind the scenes.

There are  two parts to the video processing:-

1. Video transcoding
2. Shoppable frames detection & product search.

After these parts are completed, the video is then available for editing and play on the front end.

VueStudio’s Shoppable Video Process Contains The Following Components:

• Video file – Original media file
• APIs which contain the shoppable frames and identified products – Business logic and user curated data stored per video, per account
• Video delivery – via cloudfront for optimised serving of media from edge locations.
• Rendering of the video on the front end – using the SDK which ties all the above together.

Video Transcoding

Traditionally, transcoding is the process of converting a video from one format to other formats for compatibility with other devices. However, VueStudio currently accepts videos only in .mp4 format. When an .mp4 video is uploaded, transcoding converts the video into multiple lower resolutions. This process roughly takes 3-5 minutes for a 1 minute long video. To do this, Amazon Web Services’ Elastic transcoder service is used. Once the video is transcoded, the various output formats (1080p, 720p, 360p) are stored in unique S3 buckets. The transcoded versions are then picked for serving the videos, as per the viewers’ bandwidth.

Frames Detection And Product Search

For every video, the process to identify shoppable frames happens in parallel with the video transcoding. The time for processing a video will be around 2 minutes for a 1 minute video. The edits in the video where there are scene changes, is the first thing identified. These are shown on the front end as sequential scenes in the video (Scene1, Scene 2…). This is useful to immediately identify potential scenes where there could be shoppable frames like in the image below.

Identifying Shoppable Frames

It is crucial that the most optimal frames are selected to showcase products from the video and make them shoppable, without disturbing the overall viewing experience. Shoppable frames are identified based on certain rules like blur, percentage of coverage of subject on the frame and visible duration of shoppable items (detected categories).

Linking Products To The Video Using Image Search

The shoppable frames are then used to perform an image search, where Vue.ai’s image recognition technology can identify the closest matching products from the catalog associated to the VueStudio account. Surrounding frames (to the shoppable frames) may also be used to perform the product search. Since there’s a sample catalog always available, a user can see the product search results from the closest matching items in the sample catalog. Adding your own catalog also has its advantages. Once your catalog is processed, product search results will be shown from your catalog. See image below.

A user can also delete the auto detected frames/products and add alternate shoppable frames by clicking the  “+” icon on the extreme right. This will perform a product search (personalized search) once again, for the chosen frame and find matching products from the retailer’s catalog. If the product which is automatically picked is not satisfactory, users can replace the product by searching for a different product.

Video Delivery

The serving of the video file is taken care of by AWS’s Cloudfront service using the transcoded content. This ensures that the user is able to view the video in relation to their location and bandwidth.

Tying It All Together

The Javascript SDK brings all these together for easy integration on the front end of any webpage. This same SDK is what is used by VueStudio on the landing page and on the embeddable code when it’s integrated on web pages. The SDK offers many other customization options like customising the CSS of the player and the product renderer tray. Using this option, users can customise the design of the area where the products are displayed. Refer to the documentation here to check out the ways the video can be customised.

Some examples:

 

In this manner, VueStudio brings together the possibility of creating shoppable videos of your brand and easily integrating them on your webpage. The roadmap for the next 6 months includes new features on the video tech side – with object tracking, detection of text from a video etc. and also newer interactions and designs for publishing.

We are excited to power your brand’s video marketing success story in the form of shoppable videos, are you?

Sign up for a 14 day free trial of Vuestudio here.

Read More About Automated Catalog Management Solution