Logo
Sam Eldin
CV - Resume Tools 4 Sharing Architects 2 Show Big Data Presentation Android Training Java-Unix Code Templates Interviews QA & Code

Database Replacement Using XML

To make our presentation short and sweet, if anyone searches Google looking for top issues with databases and top issues with Hadoop, Google search pages will display the followings:

     Top Database Issues      Top Hadoop Issues
       1. Growing complexity in landscape        1. Dealing with data growth
       2. Limits on scalability        2. Generating insights in a timely manner
       3. Increasing data volumes        3. Recruiting and retaining big data talent
       4. Decentralized data management        4. Integrating disparate data sources
       5. Performance        5. Validating data
       6. Security        6. Securing big data
       7. Resource utilization        7. Organizational resistance
       8. High availability        8. Production
       9. SQL injections        9. Performance
              10. Admin
              11. Concurrency

Database Storage and Data Objects Issues

Database and Data Objects
Image #1
Looking at Image #1 and tracing data in each of the cloud tiers, the following are serious issues:

1. Unmanageable number of tables and fields
2. Database Content would have: Errors, Redundancies, Duplicate, Dated Data, Inconsistencies, Not Used Stored Procedures and Functions, Duplicate Indexes, .. etc
3. Remote, batch, backup and data replication (GoldenGate) are added performance issue and security
4. Security issues in every tier
5. No consistency nor uniformity
6. Data moving through the tiers is tier and developers' dependent - no consistency or possible standardization
7. Too many conversions
8. Error prone
9. No reusability
10. Refactoring and transparency are difficult to achieve
11. SQL Injection can be added to any of the tiers
12. Slow performance
13. Difficulties in logging and tracing
14. Difficulties in audit trailing
15. Difficulties in database management - out of control

Our Database Replacement Using XML and NAS:
As we stated in the introduction of this page that data, databases and Hadoop are literally out of control with no clear solutions or answers insight. They are running in circles plus they are overwhelming the infrastructure. In short, their performance is going downhill fast. Both data vendors and the IT community need think outside the box and stop using their old database approach and tools. They should stop thinking in databases as the only data storage that can be used or worth investing into. Sadly, IT hiring managers and development teams are also stock in this old thinking, approach and technologies. We had a hard time convincing anyone (even close colleagues and friends) to view-envision any running system without tables, fields, SQL or databases.

Our approach is very simple and simply put:
Take data out the application and make it intelligent and self contained services. Use Data Access Objects (DAO) as storage within an Intelligent Data Access Object (IDAO). Such an intelligent independent service would free any running application from working with data, data security and data storage. It would also help standardize data as a service. When it comes to storing data or DAO we would simply store DAO in XML form. Such DAO to XML conversion using SOAP is straight and fast conversion. The reverse XML to DAO is also as fact and simple. No schema, any SQL calls or stored procedures, .. etc.

To see our view point, the IT community needs to look at data services as two parts:

       • Data stored in XML (text) which is the standards in sharing data
       • Intelligent data objects (our IDAO) which provide data service to any running application

The conversion of XML to DAO and DAO to XML is a straight forward process, no need to any gimmicks. Data and all its concerns is eliminated from any application. Data is stored in text files using XML (world standard). These text files can be stored on any file server and/or NAS. We have architected storage filing structure with search and fast look up design and code. The design and code are simple to develop and simpler to use. Security can be implemented using compression and encryption. Our DAO and IDAO would help standardize data usage, exchange and data communication.

Our Database REpacement
Image #2

Looking at Image #2, there is a clear line which separates data services and any running application. Our Virtual Data Services would provide both data or data answers to any running application. The fact that our Data Services is virtual, means that creating any number of its instance dynamically on the run to handle any horizontal scaling. It would be done with ease. Any real world cloud load would be handled without any issues. As for vertical scaling of our Virtual data Services would be transparent plus refactoring would not alter the external behavior.

To eliminate Our Virtual XML-DAO Convertor from bring a bottleneck and impact performance, we architected our Virtual XML-DAO Convertor with buffers and NAS storage to help speed performance plus Persistence and Memento (undo or rollback) Design Pattern.

Our Virtual File Servers do not require any anything which database servers would require. Our Virtual File Servers are architected with simple functionalities and services, no schema, no stored procedures, no SQL injections nor any complex access or usages. Both Bare-Metal servers and NAS are the only limiting factors in vertical and horizontal scaling. Communicating with the outside world, remote system or Oracle GoldenGate is simplified to file exchange. Comparing with Hadoop, anyone can see the simplicity and ease of using our Virtual File Servers.

We can state with confidence that Big Data issues are still persisting regardless of the use of the top of the line databases or Hadoop and all Hadoop's hip.

Our approach and answers to Big Data, databases and Hadoop issues are as follows:

1. Growing complexity: is handled by making data an independent intelligent service(s)
2. Growing complexity: is also done by standardizing the Data Access Objects (DAO) based on the business types and performance
3. Limits on scalability: is addressed by using virtualization to accommodate vertical and horizontal scalability and expandability
4. Increasing Data Volumes: is done using XML files to store data on file servers (local and remote)
5. Increasing Data Volumes: is done by building straight conversion from and to XML and DAO (DAO and IDAO runs in memory and XML is stored in files) plus using NAS as low cost file servers.
6. Decentralized Data Management: is addressed with the fact that data would become an independent intelligent services and with use of virtualization, there are no restrictions on data storage locations or remote access.
7. Performance: is handled by using algorithms and long integers to create short presentation of the actual data for processing speed and updates
8. Performance: is addressed by building straight conversion from and to XML and DAO. Both DAO and IDAO runs in memory as memory resident and XML is stored in files.
9. Performance: is also addressed by building virtual XML-DOA converters which would handle any load
10. Updates: will be done with stanching speed since we have created short presentation of data mention in addressing performance
11. Security: is handles by using encryption and compression to secure data storage and IDAO which handle security within the running applications
12. Security and High Availability: are done by using IDAO and DAO would be the only data objects running within any application as data services, which can be passed as function parameters
13. Resource Utilization: is done by building reusable Data Access Object (DAO) and Intelligent (IDAO) to run as data services
14. High Availability and Performance: are addressed with memory resident IDAO and DAO, virtual XML files servers and IDAP objects stored on NAS for both reusability and history.
15. Conversions: is done by creating standards data structures for data communication, parsing and conversion based on the business types
16. Production, Testing and Rollback: are done using virtual servers and IP addresses (on demand) which they aslo would help in rollback and recovery processes.
17. Integration: is done using reusable, independent, Intelligent IDAO as data services plus the use virtualization where virtual data services (virtual servers) would be created on demands.
18. Parallel Processing: is handled by independent IDAO running in parallel to handle big processes with communicating IDAO
19. Concurrency: is handled using virtual data services with target IDAO and DAO
20. SQL injections and SQL Code: are totally eliminated - No more SQL
21. ...: we have more options to present, but we do not want to overwhelm the readers


Our approach is revolutionary and doable with low cost of development and integration. Stakeholders need to look at our approach and take the inchoative by teaming up with us. As one team, we can build a pilot project perfecting our solution. We would be taking the first steps in building intelligent data services which address Bid Data, CRM, BI and end the use of databases for good.