In this application design important design consideration discussed mainly to make sure at least 1k concurrent users can use this app and also required reporting regarding review of content and dashboards. Major design objective is to scale this app to work with more users and more content.
Work flow: Ingest & Reports
Work flow: Contest creation
Request Queuing and processing:
handling huge requests . It is queue processing need to be with in the sla so that the data uploaded
can be retrieved with in 30 sec time.
This was named as "Content Agility" where Hadoop used to processing queue and quickly made consent available for end users with in 30 sec. So Data bricks can be considered for parallel
extraction of data from files and updating DB in case of app service scaling costs more for its hardware.
Important new components. are the following.
Data synchronization
Review of content can be handled by multiple admin and needs process to schedule content
and update dashboards . Signal R will make sure review data reflecting properly to
multiple admins reviewing same content.
Analytics and Reporting:
Component BUZZ will promote content to different channels based on predefined rules. Rules r stored in DB but can be moved to rule engine in future.
Eg. simple rule to promote content to main page is if any of category likes are more than 100 it will be added to the source of landing page. BFF will check for the likes and post data to Image service to update URL for landing page.
Reports r generated using data bricks and power BI, dashboards will read data from Databricks processed analytics data.
Advanced AI capabilities
Content review can be automated by using azure AI capabilities like cognitive services and others. spam detection and adult content filtering , duplicate content detection and other useful review parameters can be automated using AI capacities. which intern improve review process and accuracy of review.
Separation of Concerns
This module should be an git repo by itself if possible that will address lot of separation of concerns.
But to reuse some of the components like BFF , image viewer and upload functionality it is better to create separate library and use that one in BFF project and also make image viewer more flexible which can accept some parameters. Whole code will follow best practices of clean architecture and microservices which intern will address lot of separation of concerns.
Non Functional Requirements:
Basic infrastructure to start with will be app service for BFF, App service for widget which is part of existing UI and Service Bus for updating data to DB. To scale out to the required performance i.e 1k current Users
- App service for BFF can be made higher tier compared to Basic or Free tier. When I try to use function app which process data for 20 sec and scheduled for 5 min. total free cpu time for free tier exhausted. i.e 12 jobs per hour 24 hours running for 20 sec each job. 20*24*12 /60 min. Next app service tier may cost aorund 100$. so if we use data bricks for extraction of data and updating mongo db. It can save lot of cost on hardware.
- Async calls and using Service bus can increase decoupling of components.
- UI will post the upload data to Blob storage. Which can handle most of the calls and some bench marking can be done.
- Major processing is reading the user input and parsing the content and then pushing data to DB. This could be quite time consuming processing . Mongo DB can handle concurrent requests pretty well and the Back end component should use Tasks and update DB with bulk updates.
- Logging should use non blocking calls as well with proper tags so that session stitching quickly and retrieve required logs for analysis
- Additional component where user can give email and upload content.
- Admin can monitor the content and can delete based on some exceptions.
- As given above the content for review can use some of the AI features where
- the content can be analysed offline and mark content for ready for review.
- Using AI components will be costing bit more compared to manual work but
- can automate lot of review process.
- No user data will be saved at this stage as User login functionality is not attached to
- private data . This will be major enhancement once complete functionality is working fine.
Detailed Design:
Workflow: REVIEW
1. Validation of email if possible one time password.
2. Save file content and user details .
3. Validation of Data using AI services and manual services
4. Validation rules implementation.
5. Promoting data based on criteria.
6. Review UI
7. Enhance BFF and make it more modular by creating library for feature.
8.Analytics and dashboards using Data bricks.
9. Loggin in to Azure.
10. Major AI services including
Getting content description like image related data e.g. description..
Getting copy rights of the content.
Scanning for adult content.
Checking duplicates .
Separating premium and well classified data.
11. Implementing approval stages for the content from ingest, review, final .
12. Module should be pluggable into system so separate Data base, separate
angular component and separate function app and service bus. Only BFF
will be shared component from the talash.azure.com.au components.
No comments:
Post a Comment