LINQ And Interpreter Pattern : A tale of datasources

In a service oriented (SOA) environment, larger systems communicate/handshake with one and other with different varieties of data and formats.

These systems leverage different Data formats, contracts and pure XML (The RESTFULL way) hosted in a multi - platform environment.

Recently, in the real life, i came across a situation that i needed to fetch information from different data sources. The consuming application(s) does not want to know where these data's are being imported from and the producing application does not want to let go of the secret of the source to data. All it is concerned about is that the format must be consistent wherever it may be coming from.

I am supposed to fetch data from the following sources :

1. XML stored in a file system.
2. DataSet from another WebService
3. Service contracts from another service.

At first, this seems to be a mundane tasks, because you may have to create different code for different datasources and different translation for those datasources because an xml data is not the same as object data until we have a translator, we would not get anywhere near what we want to achieve.

First thing that comes to my mind was to leverage the Linq API and architecture to archive the above problem, how do i program my WCF service so that it will be extensible to other datasources, how do i ensure that the translation is not ambiguous (hence finding a better approach and framework that is consistent enough to do the job of translating data coming from different sources.

The Proposed architecture

I dig my hands into the dirty chests of patterns, i traveled from Gang of four to Microsoft pattern, i moved from different community, but there seem not to be a proper way of ensuring a consistent data translation strategy. After sometime, of digging, i eventually choose to use the interpreter pattern (Since it is used to interpret sentences in language elements). The reason for this choice is because i need a consistent language lexical structure that i can use for different data sources (as explained above) and Interpreter pattern seem to be the best choice to define my language grammar.


In this example, we want to interpret a Customer detail data structure coming from different sources, here is the C# structure of our Customer Poco class.

public class Customer
{
public string FirstName { get; set; }
public string LastName { get; set; }
public string Address { get; set; }
public int Age { get; set; }
}

We want to be able to translate several data sources to the Customer class defined above. This example will focus on the following data sources :

1. XML
2. Database
3. DataSet
4. object

If we are familiar with the Linq API, we would know that it supports for language elements across the named data sources above.

Interpreter Pattern Structure

The base interpreter is the interface to which we will use to interpret our language elements, because we need not know about the child classes or how they do their interpretations. The code snippet below is our base Interpreter.

public abstract class CustomerExpression
{
public Customer Customer { get; set; }
public abstract void Interpret();
}

The code above defines the CustomerExpression class and this class will define the structure of its child classes, the following child classes will be created :

1. CustomerXMLExpression
2. CustomerSQLExpression
3. CustomerDataSetExpression
4. CustomerObjectExpression

I will be discussing the first one only which is the CustomerXMLExpression. The code below is the snippet for the CustomerXMLExpression :

public class CustomerXMLExpression : CustomerExpression
{
public override void Interpret()
{
Customer = TranslateCustomer();
}

private Customer TranslateCustomer()
{
StringBuilder customerXML = new StringBuilder("XML Data");
XDocument customers = XDocument.Parse(customerXML.ToString());

return (from customer in customers.Descendants("Customers")
select new Customer
{
FirstName = customer.Element("FirstName").Value,
LastName = customer.Element("LastName").Value,
Age = (int) int.Parse(customer.Element("Age").Value),
Address = customer.Element("Address").Value,
}).FirstOrDefault();
}
}

We can use the same idea for the rest of the remaining data sources, for example we can search each data sources as follows :

CustomerExpression expression = new CustomerXMLExpression();
Customer customer = expression.Interpret();

CustomerExpression expression = new CustomerSQLExpression();
Customer customer = expression.Interpret();

CustomerExpression expression = new CustomerDataSetExpression();
Customer customer = expression.Interpret();

CustomerExpression expression = new CustomerObjectExpression();
Customer customer = expression.Interpret();

There is a long way you can go with this practise because it makes you to centralise your code and hides several dirty works away from developers.

Enum can now have method

Thanks to Extension method.

Have you ever programmed against enum in the .NEt framework. Have you realized that the .NEt enum is not as efficient as other enum living in other platforms like the Java Land. Still the .NET enum is suffering from advance ways of using constants. An .NET enum cannot even have a method (Thats how bad it is) this is referred to constants specific method in Java.

But with the method extension in the .NET framework 3.5, we can pro grammatically extend an enum with methods that makes us think that they are actually defined within the enum scope.

public enum MyEnum
{
Name,
Age,
}

public static string RealName(this enum myEnum)
{
return Enum.GetName(typeof(MyName), myEnum);
}

MyEnum enn = MyEnum.Name,

enn.RealName();

Pragmatic Approach To development. (Developing for Today and Tomorrow)

Recently i have been engrossed in a project that will contribute to the way we develop system rapidly. This project is an ORM framework that maps the impedance mismatch between the relational world and the object oriented world. I must confess its one of those projects that does not have a finished date, its a project that will continuously apply the current trend in our no destination journey of system development. What i meant by no destination is that software evolves overtime and there is no promise land, is either we re-invent the wheel and make it a better wheel or we introduce a new form of practices or complexity.

I started on this project out of frustration because designing data access frameworks per the project isn't a good development practice this can lead to inconsistent development methodologies and framework will be prone to errors. In my software development lifetime i have worked on several real world projects in their hundreds, and i understand fully well what fits well in what area. Dont get me wrong, i do mistakes too, but i accept these mistakes immediately and i make it my strength. So, over the years, i have struggled to find a long lasting solutions to poor and repeated programming approach.

Develop For Tomorrow
One of the greatest software practice i have learn't is not letting requirement to control the software but letting the software development process think ahead of what the requirement maybe in the nearer future. This is a proactive approach to developing a scalable and reliable applications that will cut across different environment and requirements.

Most of us, developers allow software requirements to drive and predict the architecture of our software for us, so when requirement changes, we need to change heavy chunk of our code. To me thats a very bad approach and it could be very costly to organisations that are practicing such approach. Where is separation of concern, where is pragmatic development.


Requirement Volatility does not mean software volatility
Feasibility checks should be done on requirements before the changes can be made. Even without full requirement, software should be built with adaptability and reconfigurability in mind, this will ensure that the volatility nature of requirements does not affect system development.


Separate as many concern as you like
Recently, in Microsoft (MSDN) Forum, somebody asked a question about separating the a UI framework or layer from a data layer, many suggestions where given to this question, some say use DTO (Data transport Object) some say use interface, the bottom line is use something that wont have adverse effect on your codebase even when the underlying structure changes, be very pragmatic and scrutinize the kind of changes that are anti pattern, we are not in the procedural programming ERA, we are developing in the world of Objects (Think in objects).

Beware of hard-coding
One of the major setback of good programming practices is hard coding. When developers hard code, it shows how vulnerable our codes can be, and how it is difficult to maitain and sustain. I wonder why we will leave a pragmatic approach to development (Even if it takes time, lets do it right). So never hard code, instead create constant file or xml or use enum.

Rapid Entity Framework: A new persistent framework

The Object Oriented Paradigm is shifting massively because coupling relational semantics into our fine grained object oriented programming designs is becoming unpopular in todays enterprise needs and domain engineering. Because of this reasons and many more, object oriented platforms is already taking a big quantum leap since the impedance mismatches between OOP and Relational has made programming against relational database an Herculean task for most development efforts.

The industry is filled with many framework's that contribute to this change, and "Rapid Entity Framework" is one out of many that will solve most of the mismatches that we face in the persistence industry today. Rapid Entity Framework is an open source framework that was discovered by a motivation to really solve the divide problem between relational DB and object oriented concepts.

With rapid entity framework, you can think about your business domain rather than columns and rows. Its a wise design efforts to really segregate applications according to their duties, "what they do", this segregation methodologies applied by industry today, should be applied to database and object (Because they are different concept). A database is data centric, and should only contain data and any data manipulation semantics. While on the other hand, an object oriented architecture see object as real life artifacts.

Visit Rapid Entity Here

How to do the laziest loading in C# and SQL Server (Part 1)

Mapping POCO (Plain Old CLR Object) to relational database could be a bit messy and a degradation of our application performance in terms of relational loading. There are several types of relationships in the database which we try to simulate with the object oriented language, these relationships are as follows :

1. One - To - One (one row to one row)
   
2. One - To Many (one row to many rows)
   
3. Many - To -One (many rows to one row. Inverse of the One - To Many)
   
4. Many - To - Many (Many rows to Many rows).

When we use the above in OOP, we come up with Aggregation, inheritance, composition, containment etc. But the way the relational data is structured and stored is different from the way data is stored in the OOP world. Data are stored on the disk in the relational world while data are stored on the stack/and heap in the CLR (Common Runtime Language).

So, to maximize the usage of data when we are mapping relational to object oriented, we need to use a pattern called "Lazy Loading".

What is Lazy Loading.
Simply, lazy loading can be described as an approach for loading objects or data when we need them. This helps to save CPU cycles because we do not intend to load what we do not need, thereby we free our heap from unessary usage, and the almighty deallocator, the Gargbage collector can go to sleep. Simple lazy loading example can be described as follows :

public IList Customers
{
get{
if(null == customer)
customers = LoadCustomers();

return customers;
}
}

From the above code snippet, we are checking for null values and if lists of customers is not null, we load the already loaded lists of customers; else we load new ones. This is a classic lazy loading feature. This kind of lazy loading will still load all customers, except a stub that refers to the list that contains the data is returned. But in the case of the classic lazy-loading, we are loading everything and returning everything. (Thats Bad, isn't it).

The Concept of Laziest Loading

Let us assume the following relationships between two objects, Customer has many Orders . The relationships that we described above is a One - To - Many scenairo, One customer to many orders, the following class diagram and code depicts the relationship :

The relationships in the diagram says one instance on a customer class will have multiple instances of an order in its relationship bag. And one instance of an order, will have one instance of a customer class. Thats it.

The Perfect Solution

In the classic lazy loading scenario, we are loading everything once it is required, even if we do not intend to use everything in the list, we end up fetching objects into the list which they will all be allocated memory space on the heap. unwanted objects made its way to the CLR heap, thats not what we bargained for.

What if we create our own special list and a class that implements IEnumerator for a special case of doing the laziest loading. The following diagram depicts our new concepts of laziest loading of data structure using our custom built list (implements from IList) and custom iterator, inplements IEnumerator : The diagram below represents the concept of the laziest loading and relational to object transformation, using datareader and our custom enumerator class.

Looking at the diagram, we will realize that there are three sections, the Custom enumeration section, the Data Reader, and the the relationship sections. This sections mapped to themselves properly to achieve the laziest loading i have been talking about.

Datareader mapping with Custom Enumerator
When you create a custom enumerator, you will implement the IEnumerator interface, which will give you various method and properties to override, but in this article, i will be talking about three methods that are very useful in our lazy loading scenario. The idea is to lazily load objects from the database when they are iterated instead of when the whole list is called. For that to happen, we need an open data reader that we need to read when the GetEnumerator method of the list that use's this Enumerator is called.

Enumerator MoveNext Mapped to DataReader Read

When there is a call to our List (Probably you need to create a custom list that implements IList, and returns our custom Enumerator). When in a froeach loop, an iterator can know if it has data to iterate with the call to movenext method. And since we are fetching from the database, we need to call the Read method of our DataReader here, this will ensure that our dataReader moves it cursor to the next row in the list. This can also return false to indicate that there is no more row to read.

public bool MoveNext()
{
return dataReader.Read();
}

Current and DataReader Get[Type]
Enumerator Current property used to translate DataReader Get[DataType] to our domain or entity object. For example, let us look at the code snippet below :

public object Current
{
get
{
ProductOrder order = new ProductOrder();
order.OrderDate = Convert.ToDateTime(dataReader[1]);
order.OrderName = Convert.ToString(dataReader[2]);

return order;
}
}

Enumerator Dispose Method (If you implement IDisposable)

The disposed method is fired when you exit the foreach loop, or you read the list to the end, or you break from the list. It is a wise idea to close the datareader in the dispose section, and before you close the datareader, it is wise to keep moving the datareader till it gets to its last (In case the loop was exited with break statement). The connection shouldnt be close here, because you may be using it for other database operations (Especially if you are using MARS (Multiple Active Result Sets) new feature in "SQL 2005 YUKON". The following code snippet, depicts the kind of stuffs that you can do with the dispose method :

public void Dispose()
{
while(dataReader.Read())
{
//Do not do anything again here. This is called when you exit the loop
//By end of loop or break statement.
}

dataReader.Close();
}

This is just the beginning, so watch out for part two when we create lazy loading handlers delegates and when i introduce you all to Ghosting or Dynamic Proxy in C# using IL (Intermediate Language). Thanks.

You can now find the Dynamic Proxy Part 1 here

Validating Domain Objects

Introducing Domain Engineering :
All OOP Languages allow users to create reusable components that cut across different domains in any business context. These components becomes an reusable asset to the practicing organization. It is called cross cutting concerns and also referred to the art of grouping objects/data's into families of system (Domain Engineering). The system families are distributed across application domains (With common variabilities and commonalities) within the same organization and/or entire world. The paradigm shift from application engineering to domain engineering gives us the gains of building reusable families of systems.

What are family of system's?
A system family is a library of components that are easily configured with other systems. In .NET a system family could be an .NET assemblies, and in JAVA it is the jar files which contains domain artifacts. When building domain objects we need to ensure that they represent what they are so as not to over engineer a system that will later on back-fire on our investments.

Let us assume the following scenario of a banking system : We have Customer, Account, Statements etc.

If we need to implement the above banking domain issues in C#, we will be modeling them each as C# POCO (Plain Old CLR Object) objects. The following diagrams forms what we would be implementing as our domain artifacts :

You can see the relationships that these object holds to themselves. An Customer object has many account. An customer object also have many statements to one account. This is a simple design of a banking domain system. If we are to create these classes, and add them in a single assembly (JAR file in Java), then, they become reusable across application domains and interfaces.


Mapping domain objects to relational database

The advent of the Object Oriented Relational Mapping has finally come, when the bigger guys in the OOP language community already have the framework interwoven with their runtime systmes. The two main OOP player of today are the .NET framework and the Java Runtime. The .NET framework has introduced the Entity Framework, while the Java people introduced java persistent framework (JPA). The mismatch between relational database and object orientation will soon be over.

When we map components or domain object to a relational database table, we are abstracting the relational semantics away from our code, which is a better thing to do. The following is the Customer code of the banking domain object in the above diagram.

Note the Customer class in the code below inherits from the BaseModel class, which is an abstract class that marks that our domain object can be validated. Note in your implementation, you can use an interface.

public abstract class BaseModel
{
public abstract bool CanValidate { get;}
}


public class Customer : BaseModel
{
private List <> statements = new List();

public List <> Statements
{
get { return statements; }
set { statements = value; }
}
private List <> accounts = new List();

public List <> Accounts
{
get { return accounts; }
set { accounts = value; }
}
}

The Domain Validation Rule Class
Since our domain objects are our relational table representation, we need to ensure that there is a valid object oriented validation before the state of our domain object hits the back end. What is a validation rule. In this example, domain rules are created as .NET attributes, and they will be used on properties of domain objects. For example of some domain rule, here are two types of rules :

Max Length Rule This rules checks the maximum length of a data in a property of a domain object. It inherits from the CustomValidationAttribute abstract class, which will be used to validate any domain object that inherits from the BaseModel

class MaxLengthRule : CustomValidationAttribute
{
private int min;

public int Min
{
get { return min; }
set { min = value; }
}
private int max;

public int Max
{
get { return max; }
set { max = value; }
}

internal override bool IsValid(T value)
{
if (null == value)
return false;

string text = value.ToString();

return (text.Length >= min && text.Length <= max); } }

NotNull rule : This rule checks if a property is null.

internal class NotNullRule : CustomValidationAttribute
{
private object value;

public object Value
{
get { return this.value; }
set { this.value = value; }
}

internal override bool IsValid(T value)
{
return (null != value);
}
}

Let us now decorate our POCO Customer class with our validation rules. The code below show how we will decorate our domain object with attributes of our validation rule :

public class Customer : BaseModel
{
private List <> statements = new List();

[NotNullRule]
public List <> Statements
{
get { return statements; }
set { statements = value; }
}
private List <> accounts = new List();

[NotNullRule]
public List <> Accounts
{
get { return accounts; }
set { accounts = value; }
}
}

Note that we have decorated our customer class with our NotNull Validation rule. This rule ensures that the value of the properties is not null.

Creating the The CustomValidationAttribute class
This class is the base class of all the validation rules that are used in this example, and it holds the entry points to the Validate method which takes the model as parameter. The following code snippet is the full code definition of the CustomValidationAttribute.

[AttributeUsage(AttributeTargets.Property)]
abstract class CustomValidationAttribute : Attribute
{
abstract internal bool IsValid(T value);

internal static bool Validate(T model) where T : BaseModel
{
Type modelType = GetModelType(model);
List validities = new List();

foreach (PropertyInfo propertyInfo in modelType.GetProperties())
{
object[] customAttributes = propertyInfo
.GetCustomAttributes(typeof(CustomValidationAttribute), true);

object value = FindValueByRule(propertyInfo, model);
foreach (CustomValidationAttribute attribute in customAttributes)
{
validities.Add(attribute.IsValid(value));
}
}

return validities.TrueForAll(el => el == true);
}

private static Type GetModelType(object instance)
{
return instance.GetType();
}

private static object FindValueByRule(PropertyInfo propertyInfo, object model)
{
return propertyInfo.GetValue(model, null);
}
}

The code above is the full defination of the base validator class. And to use the validation strategy which i have been explaining, we need to make the following simple call which returns a boolean flag that tells us if our Domain model is valid or not .

Customer customer = new Customer();

bool isValid = CustomValidationAttribute.Validate(customer);

The code above returns a boolean that indicates if the validation operation is valid or not. Happy domain validation.

Unifying Object Oriented Platforms

The Object Oriented Programming ERA is a never ending story. New platforms are being created every day. Older platforms are considered outdated, but we see them struggling for supremacy over newer and innovative ones. When Java formerly known as OAK technology joined the OOP party in the 90's, its aim was to support consumer electronics such as remote control, PDA's etc. But unfortunately, Sun microsystems lost's the bid for the consumer electronics project, which was the motive behind java. Yet, the JAVA technology quickly moved into the internet platform, seeing that java is promising in the internet era gave birth to the Java applets (an application that can run within a web browser). Java became the talk of the development industry, little did we know that some people somewhere in Redmond street are planning to revolutionise the way we write software using OOP.

The promises that Java made

Java promised to be the platform indepent OOP that will run on any operating system, mobile phones, wrist watches etc. This promise was meet with the different types of JRE (Java Runtime) that handles the native interpretation's of the Java's Bytecode into the native language. Java did not stop there but made its way through to the enterprise and server-side programming language. When J2EE was firnally introduced into the enterprise market, java became the popular platform amongst industries. Although java's cross platform features suffers performance problems over native languages because the JVM ( Java Virtual Machine ) had to be responsible for the lower level work that would have been done by the OS if it were to be a native language like C. But JAVA loyalist did not relent on the bitter side of the java technology, they concentrate more on the better side, which have seen java as the champion from inception.

But java seems to be losing out, because C# as an Object oriented Programming language is moving as fast as possible to solve the problems of software development using OOP. Still both JAVA and C# are learning from one and other.

And there was .NET

Java solves the problem of multiple platforms, but not that of multiple languages. In this our industry that is full of several programming languages, standards and patterns. There is no silver bullet that will unify these platforms. People writing java should be able to exchange components with people writing c#. These components should interoperate without any native tweaks or call.

Microsoft understands these trouble's very well, and that is why the .NET approach is to unify programming languages that targets the .NET platform. Now at last we can all invest in any kind of languages while we still use our older systems because they integrate very well with the newer ones.

But did .NET solve the problem?

.NET is not really platform independent, because .NEt was built for the windows operating systems. But there is a MONO >> http://www.mono-project.com/Main_Page project concentrating on making .NET runtime on the Linux, Solaris, Mac OS and even windows operating system. The news is Mono sponsored by Novell is an open source project and it is based on ECMA/ISO standards, the same standard for C#.

Bringing all platforms together

Time as come for us all to stop tying our developments efforts with platforms and or runtimes, time has come for us to stop using the re-inventing the wheel approach, because we can all leverage our existing systems even if we are to move from one platform to the other.

IS Mono the answer?

Mono can run .net languages as well as java, phython, Boo, PHP, javascript and many more http://www.mono-project.com/Languages . This is the long awaited universe for the software development community. A unified approach that targets different platforms and natives. Can you imagine yourselves speaking french without learning french, thats the MONO approach. We have waited so long for some thing to bring us all together, and here MONO presents itself as an opportunity.

MONO is not really the answer. I my experience of software developments and use of various technologies to solve problems, the real problem we are facing is complexity. I am very sure there is some kinds of complexity behind the use of MONO. What about learning curve, should we throw away our over the years experience in other platforms for another platform. Do we really need an interpreter for all languages? Do we need a language converter instead?