Design Uplift VIII - Trends and Analytics

Design Objectives:

1. User should be able to see trending content as landing page content. Which also should consider user preferences.

2. User clicks processed using service bus and function app to scale up the processing to more  clicks and also prioritizing user actions and if required process them offline. It can enhance scalability of processing User clicks.( likes, delete, comment , share etc)

3. Data bricks to bring near real time analytics and trends. 

4. Signal R for real time notification of content.

5. Spam detection done by audit user clicks and then  promote content or block content.

6. Azure AI  for creating prompts and getting more user interaction to define prefences instead of static list of preferences.

So above addition  of technologies can scale up the application to more number of concurrent user, can trigger back ground processing of data if needed and provide real time update to user regarding trending data. ,  which should help not  scaling  hardware  horizontally  incremental way as users/user clicks increase and increase cost of infrastructure. We can look in to more details .

Lot of samples and examples mentioned before are mostly used in my web sites. ( talash.azurewebsites.net  or TalashPDFDrive or other)  This time I am also planning to create separate repo which can be used by others and also for my web sites. Should be good attempt to create some thing generic and open source. Again Talash will be a platform byitself with quite generic components to process all types of consent and advanced functionality like content promotion and spam detection.

Technical Challenges:

1. Getting near real time trends will be processing data real time using data bricks. Demo app need not show near real time but design will explore details required to achieve.

2. Creating prompts for collecting user preferences where it can query users to get preferences and related information e.g. if     some one likes jokes and further checking if they like marriage jokes or pub jokes.

 This is one area where IT exploding with ideas now a days. I went to ecommerce web site and want to change my delivery address as I have given wrong suburb. So no need to call some one and no need to write mails or edit order which probably no ecommerce site can provide edit feature without a bug. So what is happening is just type i want to change my address then next it asks order id, then it authenticates and send email to customer care. More or less above prompt should have saved at least 2 years of development needed for ecommerce site to achieve same result.  So another hard R&D areas and will be technically challenging to design some thing efficient. there will be (Blood .. my famous quote) lot of learning by achieving tis one.

3. Using signal R effectively to update real time data and getting more engagement fro Users. It is more 

    UI / Ux challenge. I saw mixing real time updates with cache design has its own challenges. So smart invalidation of cache required.

4. Getting microservices design properly so that each service are modular and scalable. It is another challenge when code base and number of services increase and it involves addressing common code for all services vs  communication with in microservices.  As micro services stands they should be able

to work independently But it is having different patterns to communicate properly and avoiding  duplicating functionality.

5. Making sure no scalability issues in processing. Using async as much as possible and other aspects.

6/ Biggest design challenge I will explain where  building widgets directly out of data bricks.  This one interesting area where lot of engineering evolved in decades. Just by showing static report to dynamic dashboards are two extreme use cases where technology and design decisions can be more complex.

At the end of the day data tools (ETL and Reporting) always show  better way in delivering smart widgets compared to any smart coding to show trends and reports.  Now combining AI features in data tools  taking them much ahead in game.  so 80% of the widget logic will be build at the place of data rather than fetching raw data and building widget. I will deal with simple usecase whre trends are processed and combined with user preferences. So I can get widgets for above to plug in the UI like trends category wise or  charting  processed ETL data and showing them as widget on UI.

Above list will grow to more than 25 as it was developed , even small function app deployment took some time and the reason is pretty simple that .azurefucntion could not be found error. Finally it was given that files with . cannot be part of compressed deployment artifact in Linux and then changed function app hosted on windows. Still it is technically not clear but I could make it work. So there will be challenges to address on the way.

Week1 preparation: In above design diagram User action component already designed and build with service bus and function app and saved data to mongo db. The next block to design is more analysing data and building trends and UI to enable user to subscribe and add to their preferences. As discussed before it is not going to be fetching raw data and processing in an additional component , it will be processed by Data tools and prepare response good enough to bind data to  UI quickly. Explored features of Databricks quickly and it is enhanced versions of ssis or adf , able to evaluate few data processing from mongo DB and created notebooks required to get materialized views to show some trends. 

First point to note here is using Medallion Architecture which probably very good for preparing data and reporting now a days. This one also more organized way to build additional processing units to enhance the preparation of trending data. It helps to add more checks at silver layer or get more aggregated results at the gold layer. while doing above exercise I come across two videos from data bricks demos which also are quite interesting. 

1. Ghodsi's Microsoft Ignite Demo: Databricks | Databricks

2. LakeFlow Designer: Production-Quality ETL with No Code Required | Databricks

Even though my design basically need some trending data above two videos showing much better capability of data bricks inters of building advanced widgets out of data bricks quickly. First impression  is can build prompts using vector db and RAG in 10 min and some copilot kind of features in Lake flow Designer. Previously i gone through AI search training and spend hours to build vector data and then prompts and getting end point to show that one in UI. So things are now simplified but again we need the capability to evaluate the simplicity vs efficiency. That is why we are also getting companies like Perflexity which can evaluate simplify vs efficiency of these LLMs and tools. May be few of these features will be embeded in above design. For now able to build required note books.

Design Uplift: VII - Using Design patterns Part 1

Using Design patterns is one of the core element in making any application design robust and scalable.

I will give in this article few examples which are quite generic and useful for most of web development,  These are my favorites and this one is quite  challenging skill for any architect/developer to bring best design pattern in the application development. But few patterns are not very complex which can be discussed and may be few references I can mention to understand more about it.

One of my favorite pattern is strategy pattern. As shown below just there many features  which can be selected based on the Execute strategy inputs.  There are many other patterns are very useful in  apps like singleton or facade. or factory.. and others. I need to start with some thing. One  interesting example when I am learning this design pattern is one of Ecommerce product has a folder. If we move any code in that folder ex show recommendations function to that folder feature will be enabled  in that ecommerce site . 

As it is open source product I could go though implementation and finally saw the following code implementation but done exceptionally well.

There are many interesting facts about design pattens. In one product the return object / structure  from Facade layer is BOject and company name is Business Objects. After couple of days of working with that code i realize that it is best way to implement Facade. So we can go though lot of these implementations one by one.

In my project talash.azurewebsites.net User action processing implemented using strategy.  Any user action from different web clients will post data to service bus. till that point it is common code.  When processing different user actions like "Like", "delete" "comment" etc. There are many common tasks like logging , calculate rank and other things which are common to assets ,  so they are part of base strategy class and the only code functionality where  other than updating db will be  passed to the stagy derived class and executed which is very specific to User action. 

Even though it looks like just switch statement on the user action but when code organized in to above class diagram it is much cleaner and modular. User action processing will be dynamic addition just by adding the processing logic required for new action.

Let us look at Builder pattern which is creational pattern and mostly used as well.

 A strategy pattern is use to perform an operation (or set of operations) in a particular manner. In the classic example, a factory might create different types of Animals: Dog, Cat, Tiger, while a strategy pattern would perform particular actions, for example, Move; using Run, Walk, or Lope strategies.

In fact the two can be used together. For example, you may have a factory that creates your business objects. It may use different strategies based on the persistence medium. If your data is stored locally in XML it would use one strategy. If the data were remote in a different database, it would use another.

same way many operational patterns depends on deletgate

Several design patterns use delegation. The State (338), Strategy (349), and Visitor (366) patterns depend on it. In the State pattern, an object delegates requests to a State object that represents its current state. In the Strategy pattern, an object delegates a specific request to an object that represents a strategy for carrying out the request. An object will only have one state, but it can have many strategies for different requests. The purpose of both patterns

is to change the behavior of an object by changing the objects to which it delegates requests. In Visitor, the operation that gets performed on each element of an object structure is always delegated to the Visitor object.

So some times design can use multiple patterns for depends on use case.

Above explanation given just to differentiate builder pattern with strategy pattern. some use cases we may implement both together as well. So now we can focus on Builder pattern which is very common creational patterns used. 

Here is classic example of builder pattern

class MealBuilder:
    def __init__(self):
        self.builder = Builder()
    def prepare_veg_meal(self):
        self.builder.build_part1("Salad")
        self.builder.build_part2("Vegetable Curry")
        self.builder.build_part3("Rice")
        return self.builder.get_product()
    def prepare_non_veg_meal(self):
        self.builder.build_part1("Chicken Soup")
        self.builder.build_part2("Grilled Chicken")
        self.builder.build_part3("Chicken Tikka")
        return self.builder.get_product()

The Builder Pattern is a creational design pattern used to construct complex objects step by step. It will be complex to just doing every thing in constructor to construct an object. By following builder pattern it is possible to depend on few methods of the builder to do construct object. The flow and methods used to build object will differ. 

Above example is simple one. But complex objects can be build 

Pizza pizza = new Pizza.Builder(12)
                       .cheese(true)
                       .pepperoni(true)
                       .bacon(true)
                       .build();

Builder's setter methods return the Builder object they are able to be chained. Pizza object construction done using builder class.

There are many practical examples we come across for this pattern.

NET StringBuilder class is a great example of builder pattern. It is mostly used to create a string in a series of steps. The final result you get on doing ToString() is always a string but the creation of that string varies according to what functions in the StringBuilder class were used. To sum up, the basic idea is to build complex objects and hide the implementation details of how it is being built.
Sql Query Builder.  API like fluent also uses above pattern to build the fluent API and pass different values to construct .

I like the following summrization of creational patterns 

 

L Let us look in to  more complex use cases and some of important aspects we need to consider while using these creational patterns.