Sam Eldin |
CV - Resume | Tools 4 Sharing | Architects 2 Show | Big Data Presentation | Android Training | Java-Unix Code Templates | Interviews QA & Code |
|---|
|
|---|
|
Virtual Integration
What is Integration? In simple terms, System Integration is: The process of integrating (incorporate, combine, mix) all the physical and software components of an organization's system. The physical components consist of the various hardware systems such servers, computers and IO devices. The software components consists of data stored in databases, software, user interfaces and applications. Why integration is needed? The goal of system integration is sharing resources, reduce development cost, improves productivity and the quality of their operations. Integration helps various IT systems communicate which speeds up information flows and reduces operational costs. Reality Check: The best way to present the challenges of integration is looking at our Virtual Intelligent Commercial Trucking Projects (ICTP - http://intelctp.com/) which is a virtual cloud system servicing truck drivers, owner operators, truck training schools, small-midsize companies, commercial trucking business, federal and states governments, insurance companies and all truck business affiliated. All the participants may not know nor care about the rest of ICTP users or the service providers in this cloud system. All the system users would be sharing the same data, cloud services, security, hardware and the rest of system components. The core of ICTP virtual could system are: 1. Bare-Metal 2. Data 3. Services 4. Communication-interfaces 5. Security All the ICTP users must have the illusion that ICTP is a dedicated system to their needs. All behind the scene operations are transparent. Transparency refers to “the ability to allow the passage of light so that objects on the other side are clearly seen.” Transparency means invisible. In computer software, an action is transparent if it takes place without any visible effect. Transparency is usually considered to be a good characteristic of a system because it shields the user from the system's complexity or development side effects and modifying system code. The reality is that integration is very complex processes which consists of layers or levels. Integration level can be: 1. Code 2. Hardware 3. Software 4. Data 5. Services 6. APIs 7. Technologies - new technologies and old technologies 8. Communication 9. Coordination 10. Staff or personal 11. Silo Mentality 12. Internal and external 13. Internet, intranet and extranet 14. Performance 15. Security 16. Scope - Goals 17. Culture - languages Integration Issues: The following is a list of possible integration issues without any further clarifications on our part. Readers are welcome to email us their thoughts and comments (Sam@SamEldin.com) : Data: 1. Data Issues 2. Data complexity and conversion 3. Big data handling issues 4. No simplified information exchange 5. Data access issues 6. Data security 7. Ownership Communication: 1. No Clear communication 2. Lack of clear communication and responsibilities 3. No Coordination 4. Timing - on/offshore Attitude: 1. No Coordination 2. Unwillingness to outsource various operations to a third party 3. Disagreement from partners on where functionality should reside 4. Silo mentality Services 1. Security issues 2. Performance issues 3. Using different technologies 4. High cost of integration 5. No common API standards 6. Spaghetti integration 7. Connectivity issues 8. Maintenance issues 9. Scalability Issues 10. Mismatch Business Units 11. No Technical support 12. No Proper Training 13. Difficulty finding good talents 14. Access time and support Documentation 1. No documentation 2. No Training or online training 3. Poor online documentation and support Management and Integration Tools in the Market: There is a lot integration tools for data, cloud, BI, B2B, Big Data, applications, API management and middleware (Legacy system). Despite the fact that big banks and large business have their fully staffed development teams, but their management rather use outside vendors or outsource their integration. At the same time, integration vendors promise the world, but they tie their clients with their cost, hosting, frameworks and data storage. Their support may end up being a costly ordeal or a trap. Big institutions' management choose to transfer the risks, have such trap as a escape goat and at least they have someone to call for help. It seems that when it comes to integration, these institutions are not willing to take any risks developing their own homegrown integration platforms. Our Questions: • What is Virtual Integration? • How to develop homegrown virtual integration platforms? • What is the cost of developing such platforms? • How to address ownership, responsibilities, no resource sharing and lack of cooperation? • How to address the Silo mentality? • How to address scaling? • How to maintain and document homegrown virtual integration platforms? What is Virtual Integration? Virtual means that the running service or software object only exists in the computer or server memory and no physical presence nor real existence. Virtual integration is providing software services or interfaces which are only running on virtual servers. For example, a bank customer would be able to access his/her business loans, checking accounts, savings accounts, debit and credit cards and other bank services as if the bank customer is directly access the bank computer system. The reality could be that the customer is accessing a virtual server which has all the required software components to deliver the banking services. The power of virtual integration is that a virtual integration system would be able to create any number of virtual servers to handle the bank transaction load promptly and securely. Adding to the power of this virtual integration is the flexibility where these services could integrated from the bank main headquarter system, a Legacy system, a third party vendor, a remote banking services or other services which are not part of the bank. An example of other services could be ordering personal checks. This would be integrated in the bank virtual server. This would provide the customer with check ordering service without going to checks order company's site which may not be secured. How to develop homegrown virtual integration platforms? Developing a homegrown virtual integration platform is not a small task. It is economically doable, but requires Strategies, Planning, Structure and Management. We would be covering such topics in the following sections. What is the cost of developing such platforms? Based on the size of the business and physical and software resources these business have such as bare-mental servers, OS, DevOps support and experience staff. The cost is the software development-testing resources and time. The return on investment is definitely worth it since: • Their integrated system would free them from vendors and outsourcing cost and dependencies • It would be also customized to the institution needs and not One-Size-Fit All with vendors integration • Horizontal scaling (adding more services) can be automated without any additional cost • Vertical scaling (adding more virtual servers) can be automated without any additional cost How to maintain and document homegrown virtual integration platforms? The difference between throwaway system and reusable ones is management and documentation. Management must insure quality, reusability, testing, documentation and training. No short cuts nor compromising when it comes to documentation and testing. Strategies, Planning, Structure and Management: Looking at our virtual integration from 2,000 foot view, we need to have a structure, but structure without strategies, planning and management would end up in chaos. Virtual Strategies: The main strategy is to provides integration services which answer user requests according to the user resources and capabilities. In short, we would create a virtual Stateless Request and Response. This approach would require data synchronization and buffering. It is the same as Java Servlets approach. For every request, we would create a virtual server which would be a dedicated virtual server to handle the request. It would respond back with the requested services. Simply put, when the client send a request for a services, our virtual integration system would replay with a virtual IP Address pointing to a newly created virtual respond server with the service object running within this virtual respond server. Such a virtual respond server would be totally independent without interfering with other requests or responds. It is stateless. The data storage would be handled the same way would handled synchronized objects to prevent any deadlock access and overwriting of data fields or objects. The number of running virtual servers must be managed and resources would be freed and available to other requests. This can only be achieved using virtual servers to handle every requests. Our Intelligent automated DevOps and DataOps Editors: We have architected and created a prototype for Our Intelligent Automated DevOps and DataOps Editors. They provide the following:
Planning: Our plan is a step-by-step processes with dynamic Business Rules to handle any running changes in the details of these processes. Any resources can be created virtually, but based on the history and statistics, we need to plan all the needed resources, steps-processes and the virtual tools or services which would handle the load. It must be able to free and reallocate such resources. Both number of Bare-Metal and virtual servers must have a plan of handling any load and any issues including security and data access. The plan step-by-step processes would:
Management: Management would oversee all the services and resources including security and would be capable of resolving any conflicts or issues. Our management is composed of automation management and dynamic management. The automation is all the repetitive and documented management processes with a timeline and audit trail. The dynamic part is the actual human management and decision-making which are executed on daily basis. Management would be using matrices for requests, virtual servers and virtual resources. These matrices is how management would control and manage all resources, request-responds, loads, issues and security. Logging and tracking are used to help with system performance and feedbacks. Once out virtual integrated system is developed, tested and put in production, our integrated system would be automated where system administrators would be running and managing the system with no issues. Dynamic Business Rules would add run-time flexibility and transparencies. Structure: Our attempt in this webpage is to cover basic structure and not to give a lot of details which would overwhelm and/or confuse our audience specially the non-technical ones. Our 2,000 foot view Image #1 is architected-designed to present system components, services and our virtual integration services. The main objective is to present the flow of creating a virtual server with the user requested service object running within it. This is done dynamically and real-time by allocating the needed resources. Once the request is completed, then the allocated resources are freed to be reused. 
Image #1 Existing Cloud Components: Existing cloud system would have web services, application server, firewalls, database, Legacy system, networks, data centers, messaging services, hosting services, file servers, OS, third party software, Golden Gate Bi-directional sites, active sites, .. etc. Our Virtual Integration system must be able to accommodate the existing infrastructure and software plus their existing services. Our Virtual Integration would be an added services and would not interfere with the existing flow of data and processes. Services List: Applications, API, Microservices, Web Services, Data Exchange, Legacy System, Third Party Vendors, Remote Servers, Storage Services, Remote Services, ... etc. The Request-Respond Flow: The flow starts with a user sends a service request which the system firewall would check and pass it to next the Virtual Business Brokers Cluster. User requests would be handled by a virtual proxy server which would direct the request to the proper business unit broker. Based on the request a virtual Business Unit Broker (a virtual server) would handle the request. The Business Unit Broker would use the Services Matrices to figure what is needed to be done, where to find requested service and security parameters need to run the requested service. The Business Unit Broker would also check the Dynamic Business Rules for restrictions or any dynamic or late changes or rules which would be implemented. Once all check points and permissions are completed, the broker would send the request to DevOps and DataOps to create the virtual respond server with requested service object. DevOps and DataOps would team up to handle any data exchanges or data conversion needed. The DevOps would return a Virtual IP Address to the user to access the newly created-dedicated virtual respond server with the requested service object running with the virtual server. Business Unit Brokers: User requests would be handled by a virtual proxy server which would direct the request to the proper business unit broker. Each Business Unit Broker would be handling only one business unit and all the services associated with that business unit. The broker would also evaluate and grade the service provider and its services. Virtual Data Buffer Synchronized Services: The job of the Virtual Data Buffer Synchronized Services is help speed and synchronized any database and file access. Plus it would also check any security parameters needed for access databases and filing system. Matrices and Buffers: Matrices and buffers are used to dynamically and in real-time create, buffer, track, control and manage every request, respond, allocated resources, issues, exceptions, errors and audit trail requests and users. The existing security plus our security processes are addressed and there is no room for errors. Our Dynamic Business Rules add flexibilities and eliminate further recoding of processes and components. Existing Services and New Services - Services Matrices (Services + Protocols + Security Parameters): Our Virtual Integration system must track all the existing and new services. We would be developing matrices to list all services (existing or new), their location (IP addresses), the code for executing and running these services, security parameters, required resources, errors and exceptions handling and any protocols needed to successfully use these services. Users and Access: Users Matrices are also created to track users and their privileges. The types and access of each of the users (Clients, other departments, other development teams, vendors, third party vendors, .. etc) are tracked and audit trailed. Testing: All of our virtual integrating system components are virtual including the Virtual Business Brokers Cluster. Using or DevOps Editor, we would be able to create an independent Virtual Business Brokers Cluster to test the system. This Virtual testing system runs on its own virtual servers and can be tested independently. Also the system has matrices of all runs , issues, exceptions which can be used to fix all the issues and bugs. This testing process can be repeated any number of times until the system is approved and ready to move to production. Customization: System matrices are used to create, run, track and audit trail users, services and performance. These matrices give the system the ability and flexibility to customized the services, requests, responds and creation of all the virtual components running. These matrices can be easily modified and automated. Documentation: Management is responsible in forcing the services owners, to populate documentation templates. These documentation templates would be incorporated into the automated documentation and training which are virtual services. The documentation would be also be part of our virtual integration system. Log and Audit Trail Services: We have created virtual log and audit trail system with other projects and we would be more than happy to integrate into our Virtual Integration system. How to address ownership, responsibilities, no resource sharing and lack of cooperation? Each service run in its own virtual server in its own workspace. Only the service provider is responsible for providing the service object and maintaining it. They would also use our automated templates for populating the matrices and Business Rules. The templates would be tested and modified before it is used by our integration system. Testing broker would be created to test these business owner work. The testing broker would also evaluate and grade the service provider and its services. How to address the Silo mentality? A silo mentality is a reluctance to share information with employees of different divisions in the same company. Our Virtual Integration system would give every service owner the freedom to run totally independent or provide an independent virtual server with requested service. Silo issue would disappear totally. How to address scaling? All system components are virtual and can be created in any number and anytime. The load would be the only parameter in creating and freeing these virtual servers. Questions like Who owns what? Ownership is irrelevant at our virtual system since every service owner is only responsible in providing a service object which would be running within the respond virtual server. A service owner does not need to share nor give access to anyone. The owner main responsibility is providing a service object (Applications, API, Microservices, Web Services, Data Exchange, .. etc) plus the service owner would decide on the security and who would access the service and when. Performance: There is no limit on the number of virtual respond server created or any virtual resources. Only the speed of CPU and available memory are the restricting factors. We also added the Virtual Data Buffer Synchronized Services to speed the data and file storage and access. Security: Firewalls, virtual proxy server (broker cluster), Business Unit brokers and matrices of security parameters are some of the security check points. The system cloud system has its own security system which our virtual integration system must work with and implement all the required security steps and processes. Running Example: Let us at the following scenario, once a truck driver turns the ignition key on and before start driving, the following subsystem would also start:
All the system involved with such trip are working and communicating with each other and sharing the very much the same data plus personal information and data are secured and protected. ICTP is a paperless virtual intelligent automated and integrated system which speeds the driving process and makes trips safe and secure. All the ICTP processes are performed in real-time with astonishing speed. ICTP saves time, efforts, red tapes and money. ICTP would also help prevent accidents and that would save lives and properties damages. ICTP return on the investment is worth it. |
|---|