In this application design important design consideration discussed mainly to make sure at least 1k concurrent users can use this app and also required reporting regarding review of content and dashboards. Major design objective is to scale this app to work with more users and more content.
Work flow: Ingest & Reports
Work flow: Contest creation
High level Design Goals:
1. Like other Talash web sites Talash Contest app - EasyWin will have components
- UI developed for both desktop and Mobile
- BFF to accept graphql requests from UI. This helps to in both aggregating services and also other benefits like flexibility of response data fields based on UI requirement . Which can reduce number of end points required . Apollo graphql client can add lot of inbuild capabilities as Talash has many apps so even it is possible to achieve graphql federation and sharing schemas across apps.
- Major objective is to make it as framework based on micro services so that all Talash apps can share the services quickly and reusability will be better.
- Operability point of view azure logging and using cloud for deployment can help to scale this app. Come across a tool where logs will show sql fired for that functionality and params passed to that sql Even though the code developed with ORM. that is quite innovative and reduce time to resolve issues quite quickly. These kind of tools integrations makes operability better. My favourite is serilog for logging as it has lot of integrations and data dog which can show response time of each api end point quickly. At some point I am using data dog to compare response time of each api between two releases and find out any performance issues quickly.
- Scalability wise either event driven or service bus processing should be implemented to handle high volumes of request processing wherever possible.
- Lot of analytics tools integration like data bricks , power BI helps to understand the user analytics and trends on the app and can help lot in enhancing the geo based or demographic based useful and prime content which can increase user engagement.
- Objective of all talash apps is not hosting huge content but just hosting prime content and user should be easy to grab prime content from web and can share and save for day to day use.
Innovation
Framework design to support multiple apps. Framework will have well defined API later and based on micro services .
Content classification algorithms are unique and and can quickly filter the spam. Here AI will help lot compared to previous algorithms used in youtube or other content hosting sites.
Third one is the prime content like flicker.com . It is site for photographers and people who enjoy photography. It is able to achieve with 100% accuracy. Same way Talash apps are unique in hosting
content and classifying content.
Content Sharing : Apps helps to identify interesting content quickly and share with others.
Request Queuing and processing:
handling huge requests . It is queue processing need to be with in the sla so that the data uploaded
can be retrieved with in 30 sec time.
This was named as "Content Agility" where Hadoop used to processing queue and quickly made consent available for end users with in 30 sec. So Data bricks can be considered for parallel
extraction of data from files and updating DB in case of app service scaling costs more for its hardware.
Data synchronization
Review of content can be handled by multiple admin and needs process to schedule content
and update dashboards . Signal R will make sure review data reflecting properly to
multiple admins reviewing same content.
Analytics and Reporting:
Component BUZZ will promote content to different channels based on predefined rules. Rules r stored in DB but can be moved to rule engine in future.
Eg. simple rule to promote content to main page is if any of category likes are more than 100 it will be added to the source of landing page. BFF will check for the likes and post data to Image service to update URL for landing page.
Reports r generated using data bricks and power BI, dashboards will read data from Databricks processed analytics data.
Advanced AI capabilities
Content review can be automated by using azure AI capabilities like cognitive services and others. spam detection and adult content filtering , duplicate content detection and other useful review parameters can be automated using AI capacities. which intern improve review process and accuracy of review.
Separation of Concerns
This module should be an git repo by itself if possible that will address lot of separation of concerns.
But to reuse some of the components like BFF , image viewer and upload functionality it is better to create separate library and use that one in BFF project and also make image viewer more flexible which can accept some parameters. Whole code will follow best practices of clean architecture and microservices which intern will address lot of separation of concerns.
Non Functional Requirements:
Basic infrastructure to start with will be app service for BFF, App service for widget which is part of existing UI and Service Bus for updating data to DB. To scale out to the required performance i.e 1k current Users
- App service for BFF can be made higher tier compared to Basic or Free tier. When I try to use function app which process data for 20 sec and scheduled for 5 min. total free cpu time for free tier exhausted. i.e 12 jobs per hour 24 hours running for 20 sec each job. 20*24*12 /60 min. Next app service tier may cost aorund 100$. so if we use data bricks for extraction of data and updating mongo db. It can save lot of cost on hardware.
- Async calls and using Service bus can increase decoupling of components.
- UI will post the upload data to Blob storage. Which can handle most of the calls and some bench marking can be done.
- Major processing is reading the user input and parsing the content and then pushing data to DB. This could be quite time consuming processing . Mongo DB can handle concurrent requests pretty well and the Back end component should use Tasks and update DB with bulk updates.
- Logging should use non blocking calls as well with proper tags so that session stitching quickly and retrieve required logs for analysis
- Additional component where user can give email and upload content.
- Admin can monitor the content and can delete based on some exceptions.
- As given above the content for review can use some of the AI features where
- the content can be analysed offline and mark content for ready for review.
- Using AI components will be costing bit more compared to manual work but
- can automate lot of review process.
- No user data will be saved at this stage as User login functionality is not attached to
- private data . This will be major enhancement once complete functionality is working fine.
SAAS based solutions and Pricing:
Cloudanary: Not very expensive to get quick response of content from CDN. alternative will be akamizing content.
Search API: Google search api searchapi.io is quite good option. It cost 40$ per month for 10k requests per month.
AI TOOLS: Azure cognitive services has many tools. Right now using text analyzers right now it is not quite expensive. But evaluating visual tools and their cost. Google api also provide image analyzing API like google lens. Integrating these API is quite easy.
EmailJS. it is quite good and has very easy integration of templates and send mail functionality.
It is not expensive till 1k mails.
Genderize.io: It is my favourite and costs around 10$ per month and Definity quite good API.
Dyte: Video CDN and with good api for conferencing. It has free plan till certain number of request.
Auth0: it is used for security and user management. costs 35$ . Quite good at development stage.
Right now It costs 30$ for mongo db and azure appservices. So just planning to run 3 talash web sites below 100$.
Detailed Design:
Workflow: REVIEW
Important new components. are the following.
Contest Creation
- Validation of email if possible one time password.
- Save file content and user details .
Content classification
- Validation of Data using AI services and manual services
Content filtering
- Validation rules implementation.
- Major AI services including
- Getting content description like image related data e.g. description..
- Getting copy rights of the content.
- Scanning for adult content.
- Checking duplicates .
- Separating premium and well classified data.
- Review UI
- Implementing approval stages for the content from ingest, review, final .
Content security
Content Promotion
- Promoting data based on criteria.
Contest Dashboards.
- Analytics and dashboards using Data bricks.
Non Functional Requirements:
- Enhance BFF and make it more modular by creating library for feature.
- Loggin in to Azure.
- Module should be pluggable into system so separate Data base, separate angular component and separate function app and service bus. Only BFF will be shared component from the talash.azure.com.au components.
