Digital Signage Based User Targeted Advertising

Problem and Solution

Most of the advertisements displayed in the digital screens are not relevant for the targeted audience. So,ninety percent of the advertisements displayed using the currently existing digital signages are not very effective. Also the provided digital signage solutions are expensive to afford for most of the SMEs(Small and Medium Enterprises).

To overcome these kind of issues our proposed solution consists of an upgraded version with additional hardware/software which can do User Targeted Advertising. Here our aim is while showing content that changes dynamically, when a particular crowd (male/female/children) is identified in front of the screens it will change the current content flow and show some relevant,interesting advertisement for the targeted audience that might catch their attention.

The buyer(shop owner) of the digital signage unit can upload their advertisements using the provided user authenticated web application or the mobile application. Also add some specific advertisements categorized by gender and age group. When a user in front is detected specific content will be visible to the targeted audience.

Watch Product Video

Design Architecture

Design Technologies

The digital signage control unit consists of raspberry pi 3 which acts as the heart of the system.
The Digital display is connected to raspberry pi using HDMI and the user detecting and Identifying unit is attached to the digital display. This user detecting and Identifying unit mainly consists of Raspberry PI camera module and distance sensor module.
Firebase is used as the back-end.
Flutter is used as the web/mobile app framework for the front-end development. Dart is used as the programming language where the same code base can be used to develop both android and ios applications.
As an additional improvement, a smart power supply unit which can control the digital screen on/off through the web application will be implemented.

ER Model - Database Structure

Main Entities Of The Database are ADMINISTRATION, CUSTOMER(Shop owner who is willing to buy signage unit), ADVERTISEMENT and WORK_TIME. Main relationships are REGISTERS, UPLOADS and SCHEDULES.

ADVERTISEMENT entity has a unique id as the primary key. Advertisements can be in three types.
-Image, Ticker(Text), Video

CUSTOMER table has a composite key, Customer_id along with Device_id. Having a composite key here is useful when same customer is buying more than one signage unit. Here,Device_id refers to the MAC address of the NIC of digital signage controlling unit.

1:n relationship exists between ADMINISTRATION and CUMSTOMER, beacause a customer only has one ADMINISTRATION and ADMINISTRATION has many customers. And we can see total participation in both sides because everyone should be registered to use the system.

As digital screen can be turned on and off only within the working hours because of our smart power supply unit, customer can schedule the time slots for each day. That data is stored in WORKING_TIME entity. It is a weak entity that does not have a primary key. Here, we have got a weak key 'Day' and same key can be repeated.

Simplified Circuit Diagrams

Components Needed

Digital Signage Controlling Unit

Raspberry Pi - $ 35
RPi Power Supply - $ 9
SD card - $ 10
Cooling Fan - $ 4
HeatSink - $ 1
LED Push Button - $ 4
Cover - $ 7

User Identifying and Detecting Unit

Raspberry Pi - $ 35
RPi Power Supply - $ 9
SD card - $ 10
Cooling Fan - $ 4
HeatSink - $ 1
Camera Module - $ 10
LED Push Button - $ 4
Cables - $ 5
LEDs, Resistors(220Ω) - $ 0.5
Cover - $ 7

Smart Power Supply Unit

NodeMCU esp8266 - $ 5
Relay Module - $ 3
Rocker Switch - $ 5
Plug socket & Top - $ 5
Cables - $ 3
LED, Diodes, Resistors(100kΩ) - $ 1
Cover - $ 7

Flow Chart

Data Flow

User is supposed to upload the advertisements using the mobile app or web app.
Those advertisements are uploaded into SD card in RPI board through the Web server.
Image of the person who is infront of the screen is captured by the camera module and given to RPI board.
That image is processed with trained data models and gender and age groups of the person is caculated.
According to the results, best advertisement is shown to the user.

Software Architecture

Data Flow

User can upload advertisements using the mobile app.
Advertisements are uploaded as google slides which are stored inside google cloud. Because of using google slides, we could minimize the cloud storage of our system. And user can easily create and edit advertisement using google slides.
User information is stored inside the Firebase (in Cloud Firestore).
Detected user age and gender predictions are realtime updated in firebase.
Screenly API (in RPI) fetch the correct advertisement from cloud and display in the screen.

User Targeted Advertising

Age and gender detection in our system is mainly done with Convolutional Neural Networks.
We have used the CNN models trained by Gil Levi and Tal Hassner (two Israel researchers) using caffe framework and we have used the OpenCV’s dnn package which stands for “Deep Neural Networks”.
To detect faces from images,webcam and to import neural network trained models, opencv libraries such as haarcascade,dnn packages are used in our implementation for predicting gender and age.
CNN consists of 8 values for 8 age classes (“0–2”, “4–6”, “8–13”, “15–20”, “25–32”, “38–43”, “48–53” and “60-100”) and two values for two gender clases ("Male" and "Female.")
As you can see in these pictures, this model accurately predict the gender and the age group.
Link for Github Files

Test To Check Accuracy When Multiple Faces Appears

Test

Check if image predicted correctly when there is more than one face appearing.

Test Type

Unit Testing(We did Black Box Testing)

Tool

Unit Test Library In Python

Results

Smart Power Supply Unit

Overall Design

Components and Procedure

MQTT protocol is used to design the smart power supply unit.
Mobile app (MQTT client) publishes user given messages (ON or OFF) to a unique topic for that power supply unit.
Mac address of the Node MCU is used as the topic to make the topic unique for that power supply unit.
Node MCU(ESP8266) subscribes to the certain topic and recieves user given message through the broker.
EMQX broker was used in our implementation.
After recieving the user given message, it is given to 5V relay module by the NodeMCU. According to the message, screen is going to turn on or turn off.

Hardware Design

NodeMCU (ESP8266)
AC to DC converter
Relay (5v)
Plug
Power Socket
Switch

Indicators
Resistors

Safety Factors

5A fuse is included in the plug to protect the relay module from high currents.
Heat sink is added to protect the device if it becomes overheated.
Earthing is used here to protect the user from an electric shock.

Special Concerns

If a wifi failure happens, you can disconnect the smart power supply unit by the switch and switch on or off manually.
If there is a power failure, you don’t have to worry. It will execute the last message given by the user.
Using WifiManager library, ESP8266 can be connected to Wifi router without hardcoding router SSID and password in the code.

App frontend Design...

Unit testing for app and backend

Results and Findings

cloud firestore mocks and flutter test dependencies are used for several testing in both app and the backend firebase

User entry validations performed to provide a better user experience. Invalid user inputs captured and informed properly. Added mac addresses by users are realtime synced to the menus and testing done for device adding and deleting procedures in app.

Firestore is used as the backend databse and collections are used to organize data structure of users and devices.

Mac addresses are used for document topics in firestore to identify each device. Testing perfomed with this structure for correct retrieval and deletion of devices through app by the user.

Results from tests optimized the structure of the firestore for categorizing users and also providing analytics by realtime counts of people captured for user targeting in each gender category.