Design Uplift VIII - Trends and Analytics

Design Objectives:

1. User should be able to see trending content as landing page content. Which also should consider user preferences.

2. User clicks processed using service bus and function app to scale up the processing to more  clicks and also prioritizing user actions and if required process them offline. It can enhance scalability of processing User clicks.( likes, delete, comment , share etc)

3. Data bricks to bring near real time analytics and trends. 

4. Signal R for real time notification of content.

5. Spam detection done by audit user clicks and then  promote content or block content.

6. Azure AI  for creating prompts and getting more user interaction to define prefences instead of static list of preferences.

So above addition  of technologies can scale up the application to more number of concurrent user, can trigger back ground processing of data if needed and provide real time update to user regarding trending data. ,  which should help not  scaling  hardware  horizontally  incremental way as users/user clicks increase and increase cost of infrastructure. We can look in to more details .

Lot of samples and examples mentioned before are mostly used in my web sites. ( talash.azurewebsites.net  or TalashPDFDrive or other)  This time I am also planning to create separate repo which can be used by others and also for my web sites. Should be good attempt to create some thing generic and open source. Again Talash will be a platform byitself with quite generic components to process all types of consent and advanced functionality like content promotion and spam detection.

Technical Challenges:

1. Getting near real time trends will be processing data real time using data bricks. Demo app need not show near real time but design will explore details required to achieve.

2. Creating prompts for collecting user preferences where it can query users to get preferences and related information e.g. if     some one likes jokes and further checking if they like marriage jokes or pub jokes.

 This is one area where IT exploding with ideas now a days. I went to ecommerce web site and want to change my delivery address as I have given wrong suburb. So no need to call some one and no need to write mails or edit order which probably no ecommerce site can provide edit feature without a bug. So what is happening is just type i want to change my address then next it asks order id, then it authenticates and send email to customer care. More or less above prompt should have saved at least 2 years of development needed for ecommerce site to achieve same result.  So another hard R&D areas and will be technically challenging to design some thing efficient. there will be (Blood .. my famous quote) lot of learning by achieving tis one.

3. Using signal R effectively to update real time data and getting more engagement fro Users. It is more 

    UI / Ux challenge. I saw mixing real time updates with cache design has its own challenges. So smart invalidation of cache required.

4. Getting microservices design properly so that each service are modular and scalable. It is another challenge when code base and number of services increase and it involves addressing common code for all services vs  communication with in microservices.  As micro services stands they should be able

to work independently But it is having different patterns to communicate properly and avoiding  duplicating functionality.

5. Making sure no scalability issues in processing. Using async as much as possible and other aspects.

6/ Biggest design challenge I will explain where  building widgets directly out of data bricks.  This one interesting area where lot of engineering evolved in decades. Just by showing static report to dynamic dashboards are two extreme use cases where technology and design decisions can be more complex.

At the end of the day data tools (ETL and Reporting) always show  better way in delivering smart widgets compared to any smart coding to show trends and reports.  Now combining AI features in data tools  taking them much ahead in game.  so 80% of the widget logic will be build at the place of data rather than fetching raw data and building widget. I will deal with simple usecase whre trends are processed and combined with user preferences. So I can get widgets for above to plug in the UI like trends category wise or  charting  processed ETL data and showing them as widget on UI.

Above list will grow to more than 25 as it was developed , even small function app deployment took some time and the reason is pretty simple that .azurefucntion could not be found error. Finally it was given that files with . cannot be part of compressed deployment artifact in Linux and then changed function app hosted on windows. Still it is technically not clear but I could make it work. So there will be challenges to address on the way.

Week1 preparation: In above design diagram User action component already designed and build with service bus and function app and saved data to mongo db. The next block to design is more analysing data and building trends and UI to enable user to subscribe and add to their preferences. As discussed before it is not going to be fetching raw data and processing in an additional component , it will be processed by Data tools and prepare response good enough to bind data to  UI quickly. Explored features of Databricks quickly and it is enhanced versions of ssis or adf , able to evaluate few data processing from mongo DB and created notebooks required to get materialized views to show some trends. 

First point to note here is using Medallion Architecture which probably very good for preparing data and reporting now a days. This one also more organized way to build additional processing units to enhance the preparation of trending data. It helps to add more checks at silver layer or get more aggregated results at the gold layer. while doing above exercise I come across two videos from data bricks demos which also are quite interesting. 

1. Ghodsi's Microsoft Ignite Demo: Databricks | Databricks

2. LakeFlow Designer: Production-Quality ETL with No Code Required | Databricks

Even though my design basically need some trending data above two videos showing much better capability of data bricks inters of building advanced widgets out of data bricks quickly. First impression  is can build prompts using vector db and RAG in 10 min and some copilot kind of features in Lake flow Designer. Previously i gone through AI search training and spend hours to build vector data and then prompts and getting end point to show that one in UI. So things are now simplified but again we need the capability to evaluate the simplicity vs efficiency. That is why we are also getting companies like Perflexity which can evaluate simplify vs efficiency of these LLMs and tools. May be few of these features will be embeded in above design. For now able to build required note books.