Database Migration and Integration using AWS DMS

Amazon Web Services (AWS) recently released a product called AWS Data Migration Services (DMS) to migrate data between databases.

The experiment

I have used AWS DMS to try a migration from a source MySQL database to a target MySQL database, a homogeneous database migration.

The DMS service lets you use a resource in the middle Replication Instance - an automatically created EC2 instance - plus source and target Endpoints. Then you move data from the source database to the target database. Simple as that. DMS is also capable of doing heterogeneous database migrations like from MySQL to Oracle and even synchronous integrations. In addition AWS DMS also gives you a client tool called AWS Schema Converter tool which helps you convert your source database objects like stored procedures to the target database format. All things a cloud data integration project needs!

In my experiment and POC, I was particularly interested in the ability of the tool to move a simple data model as below, with 1-n relationship between tables t0(parent) and t1(child) like below.

(Pseudo code to quickly create two tables t0, t1 with 1-n relationship to try it. Create the tables both on source and target database)

t0 -> t1 Table DDL (Pseudo code)

CREATE TABLE `t0` (
  `id` int(11) NOT NULL,
  `txt` varchar(100) CHARACTER SET ucs2 DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `t1` (
  `id` mediumint(9) NOT NULL AUTO_INCREMENT,
  `t0id` int(9) DEFAULT NULL,
  `txt` char(100) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `t0id` (`t0id`),
  CONSTRAINT `t1_ibfk_1` FOREIGN KEY (`t0id`) REFERENCES `t0` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


In this experiment, I didn't want to see just a migration, a copy, of a table from source database to a target database. I was interested more to see how easy is to migrate a data model - with Primary Key and Foreign Key relationship in place -  from the source database to the target database with zero downtime and using their CDC (Changed data Capture) or Ongoing-Replication migration option and capabilities of AWS DMS. That is, zero downtime database migration.

Here are the results of the experiment.

AWS DMS is ubiquitous, you can quickly set-up an agent (Replication Instance) and define source & target endpoints and start mapping your tables to be migrated from source database to target database with the tool. All conveniently using the AWS console.

Once you setup your replication instance and endpoints, create a Migration Task (say Alpha) and do an initial full migration (load) from the source database to the target database. Do this with the foreign keys (FKs) disabled on the target. This is a recommendation in the AWS DMS Guide in order to dump the data super fast as it does it with parallel threads, at least this is the recommendations for MySQL targets.

Then you can create a second Migration Task (say Beta) using a different endpoint, but this time with the foreign keys enabled on the target. You can do this even before your full load with Alpha to avoid waiting times. Configure Beta interface/task to run forever and let it integrate and sync the delta which occurred during the initial load. You can even start the Beta interface from a cut-off timestamp point. It uses source MySQL database's binlogs to propagate the changes. If you don't create beta interface, that is to use a different endpoint for the target with the parameter which enables the FKs, the DELETE SQL statements on the source which occur during the migration will not propagate to the target correctly and the CASCADEs to the child tables will not work on the target. CASCADE is a property of the Foreign Key.

To reconcile, to find out if you have migrated everything, I had to count the rows in each table on source and the target databases to monitor and see if it all worked. To do that I used Pentaho Spoon CE to quickly create a job to count the rows on both source and target database and validate migration/integration interfaces.

Overall, I found AWS DMS very easy to use, it quickly helps you wire an integration interface in the Cloud and start pumping and syncing data between sources and targets databases be it on Premise or Cloud. A kind of Middleware setup in AWS style, in the Cloud. No more middleware tools for data migration, AWS now has it's own. 

LEVERAGE GEOGRAPHICALLY-DISTRIBUTED DEVELOPMENT

As technology advances at warp speed, there are certain tech methodologies that will go by the wayside to make room for more advanced and efficient versions; and how development projects are managed is a case, in point. Companies in every industrialized nation of the world are embracing Geographically-Distributed Development or GDD, which has embedded itself an impressive and proof-positive IT strategy model. Outdated procedures that have been utilized for the administration of virtually any type of development project have been limited to one or several building sites. That was then; this is now.


Let’s take a look at the advantages that GDD offers:

• decreased labor expenses 
• increased availability to skilled resources
• reduced time-to-market, with round-the-clock flexible staffing

The beauty of GDD is that is allows enterprises, regardless of location, to respond to changes in business circumstances as they happen. Any feedback can be presented, instantaneously, within a global framework.

In order for GDD to achieve its vast benefit potential, major barriers that might impede an enterprise’s successes must be reduced to a minimum or entirely eliminated within the GDD strategy. It is crucial that increased expenses associated with communication and coordination logistics that occur on an international level within a globally-distributed market, be uncovered and targeted. If communication and coordination-specific expenses are allowed to flourish, the very benefits of GDD can be sorely compromised. Various challenges must be reckoned with: 1) cultural variances 2) language differences and 3) inaccessibility to time-sensitive information. These can all jeopardize the progress of distributed projects.

GDD is oblivious to location. it is an IT strategy model without borders. This allows development team-members to work collectively and cohesively within a city, across state lines or beyond continents. A site or sites might be engaged with one particular software-development venture while one or more outsourcing companies work, simultaneously, towards the projected goal. Outsourcing companies would contribute their efforts and expertise, like a fine-tuned engine, within the software’s project-development cycle. Optimized efficiency and cost savings, via structured and coordinated local or global team-work, becomes refreshingly realized.

With GDD, thorough and clear communication is established between all team members and project coordination. Business demands incorporate global-sourcing, service-oriented architecture, new compliance regulations, new development methodologies, reduced release cycles and broadened application lifetimes. Because of this, highly-effective, unencumbered communication is mission-critical; and a necessity arises that begs for a solution that has the power to:

• Provide management visibility of all change activities among distributed development teams 
• Integrate and automate current change processes and best practices within the enterprise
• Organize the distribution of dependent change components among platforms and teams
• Protect intellectual property

Track and authenticate Service Level Agreements (SLAs)

Engaging an organization to efficiently manage and significantly optimize communication among all stakeholders in the change process is a priceless component of an Application Lifecycle Management (ALM) solution. Multiple GDD locales present inherent challenges: language and cultural divides, varying software-development methods, change-management protocol, security employment, adherence to industry mandates and client business requisites. The good news is the ALM solution tackles these hurdles with ease!


Provide Management Visibility of all Change Activities among Distributed Development Teams
When a centralized repository allows for the viewing of all the activities, communications and artifacts that could be impacted by the change process, you have beauty in motion; and this is what ALM does. Via ALM, users have the luxury of effortlessly viewing project endeavors by each developer, development group or project team--irrespective of location, platform and development setting. This type of amenity becomes especially striking when one begins to compare this model-type with other distributed environments where work-in-progress is not visible across teams due to niche teams employing their own code repositories.

ALM provides the opportunity for development managers to not only track, but validate a project’s standing. A project’s status can be verified which helps to guarantee the completion of tasks. User-friendly dashboards will alert management if vital processes indicate signs of sluggishness or inefficiency.

ALM ensures that the overall development objectives will be met on a consistent basis. This is accomplished through the seamless coordination between both remote and local development activities. The ALM-accumulated data plays a crucial role with boosting project management, status tracking, traceability, and resource distribution. Development procedures can be continually improved upon, thanks to generated reports that allow for process metrics to be collected and assessed. Also, ALM allows regulatory and best-practices compliance to be effortlessly monitored and evaluated. Compliance deals with structuring the applicable processes and creating the necessary reports. ALM executes compliance strategy and offers visibility to the needed historical information, regardless of users’ geographic locations.


Integrate and Automate Current Change Processes and Best Practices within the Enterprise
In a perfect world, each and every facet of a company’s application development would be super easy; and with ALM it is. By way of ALM, companies can establish the defined, repeatable, measureable and traceable processes based on best practices, with absolute perfection. User-friendly point-and-click set-up functions enable one to create a collection of authorized processes that automate task assignments and movement of application artifacts.

ALM permits the streamlining of change management by means of its simplicity when dealing with changes and necessary proceedings. This in turn means changes can be analyzed and prioritized. The approval management functions demand that official authorizations must be secured before any changes are permitted to go forth. The ATM’s automated logging functions totally un-complicate the tracking of software changes. This is huge since changes can be tracked from the time a request is received up to the time when a solution is submitted to production.

Every member that is part of the global development team would be duly notified regarding any required assignments as well as any happenings that would have an impact on their efforts.


Organize the Distribution of Dependent Change Components among Teams and Platforms
It’s no secret that when there are changes within just one system of a cohesive enterprise, those changes can impact other systems. ALM offers multi-platform support which ensures that modifications made on disparate platforms, by way of geographically-dispersed teams can be navigated through the application lifecycle jointly. A Bill of Materials Process, or BOMP, serves as an on-board feature that permits users to create file portfolios that incorporate characteristics from various platforms. This means those portfolios can travel through the lifecycle as a unit. Additionally, some ALM solutions absolutely ensure that the parts within the assemblies are positioned with the suitable platforms at each state of the lifecycle.


Protect Intellectual Property
An ALM solution is the perfect component that allows for access and function control over all managed artifacts. Managers are in a position to easily determine and authorize any access to vital intellectual property due to ALM functioning on a role-based control system. The role-based structure means administrative operations are streamlined which permits any system administrator to bypass assigning individual rights to a single user. Additionally, a role-based system delivers a clearly-defined synopsis of access rights between groups of individuals.

Track and Authenticate Service Level Agreements
The overall project plan, absolutely, must remain on schedule while administering accountability for established deliveries; and this can be easily realized through ALM’s ability to track and authenticate tasks and processes. The ALM solution caters to satisfying Service Level Agreement (SLA) requirements within an outsourcing contract. As a result, project management is enhanced by ensuring performance of specific tasks. Optimizing the user’s ability to track emphasized achievements is made possible due to the consistency between tasks that have been assigned to developers and tasks that are part of the project plan. Invaluable ALM-generated reports will track response and resolution times; and service-level workflows automate service processes and offer flexibility. This translates into an acceleration of processes to the respective resources to meet project deadlines. The ability to track performance against service-level agreements is made possible due to the availability of reports and dashboards that are at one’s fingertips.


Enhance Your Geographically-Distributed Development
As stated, ALM is beauty in motion; and aside from promoting perfected-levels of communication and coordination, it utilizes management strategies designed to plow through any obstructions that have the potential to compromise success. ALM’s centralized repository is purposed to present multiple ideas, designs, dialogue, requirements, tasks and much more to team-members who would require or desire instant access to data. Development procedures and tasks can be precisely and efficiently automated and managed due to ALM’s cohesive workflow capabilities. Vital intellectual property, all of it is embedded and safeguarded in a central repository. Due to this caliber of reinforced protection, loss and unauthorized access is null and void. When remote software development is in-sync with local development, project management becomes seamless, highly-coordinated and error-free. Integration of the monitoring, tracking and auditing of reports and dashboards means management can successfully satisfy project deadlines. It would behoove any enterprise who wishes to reap the rewards of GDD to fully embrace ALM as its solution, it is truly mission-critical.


Application Lifecycle Management Solutions
Application lifecycle management programs are able to easily deliver a plethora of enterprise software modifications and configuration management facilities. ALM solutions have the ability to support the needs of multiple groups of geographically-distributed developers. Business process automation services, designated to automate and enforce on-going service delivery processes throughout enterprise organizations, is a vital component of ALM solutions. Within those groups of geographically-distributed developers, the product continues to reveal the magnitude of its talents since it: targets permission-based assignment and enforcement services, caters to role-based interfaces which allows support for developers, software engineers, project managers, IT specialists, etc, delivers enterprise application inventory-management services, oversees and coordinates large software inventories and configurations, guards user access, manages differing versions of application code, supports the existence of concurrent development projects, coordinates a diversity of release management facilities.

Guest blogger post:

Mike Miranda is a writer concerning topics ranging from Legacy modernization to Application life cycle management, data management, big data and more.

Oracle Apex 5.0 and APEX_JSON

How many lines of code does it take to make a web service call? Answer: 39

That is how many lines of PL/SQL I had to write in Oracle Apex 5.0 to make a web service call to an external API.

I used Adzuna's REST API to retrieve the latitude and longitude and the price of 2 bed properties for rent in a specific location in UK. The API returns JSON which the APEX_JSON package is able to parse easily. Adzuna is a property search engine which also provides aggregate data for properties in various countries around the world.

I think the native APEX_JSON package in Oracle Apex 5.0 is very useful and makes integrating web services to your Oracle Apex applications very easy. Here is application I have created in matter of hours which shows you average rent properties in a location of your choice in UK.

Here is the link for the app: http://enciva-uk15.com/ords/f?p=322:1

And here is the code:

If you want to run this as is in your SQL Workshop, make sure you replace {myadzunaid:myadzunakey} in the code with your adzuna app_id and app_key which you can obtain from the Adzuna website https://developer.adzuna.com/ as I have removed mine from the code. They also have a very good interactive api documentation here http://api.adzuna.com/static/swagger-ui/index.html#!/adzuna

create or replace procedure get_rent_data(p_where in varchar2, p_radius in number, p_room in number)
is
v_resp_r clob;
j apex_json.t_values;
l_paths apex_t_varchar2;
v_id varchar(50);
v_lat decimal(9,6);
v_lon decimal(9,6);
v_rent number(10);

begin
-- http housekeeping
apex_web_service.g_request_headers(1).name  := 'Accept'; 
apex_web_service.g_request_headers(1).value := 'application/json; charset=utf-8'; 
apex_web_service.g_request_headers(2).name  := 'Content-Type'; 
apex_web_service.g_request_headers(2).value := 'application/json; charset=utf-8';

v_resp_r := apex_web_service.make_rest_request 
      ( p_url => 'http://api.adzuna.com:80/v1/api/property/gb/search/1' 
      , p_http_method => 'GET' 
      , p_parm_name => apex_util.string_to_table('app_id:app_key:where:max_days_old:sort_by:category:distance:results_per_page:beds') 
      , p_parm_value => apex_util.string_to_table('{myadzunaid:myadzunakey}:'||p_where||':90:date:to-rent:'||p_radius||':100:'||p_room||'') 
      );
-- parse json
apex_json.parse(j, v_resp_r);


-- start looping on json
l_paths := apex_json.find_paths_like (
        p_values         => j,
        p_return_path => 'results[%]',
        p_subpath       => '.beds',
        p_value           => '2' );
        
for i in 1 .. l_paths.count loop
       v_id := apex_json.get_varchar2(p_values => j, p_path => l_paths(i)||'.id'); 
       v_rent := apex_json.get_varchar2(p_values => j, p_path => l_paths(i)||'.price_per_month'); 
       v_lat := apex_json.get_varchar2(p_values => j, p_path => l_paths(i)||'.latitude');
       v_lon := apex_json.get_varchar2(p_values => j, p_path => l_paths(i)||'.longitude');

-- debug print values to page
 htp.p(v_id||'-'||v_lat||','||v_lon||'Rents : £'||v_rent);

end loop;

END;

Five Questions to Ask Before Purchasing a Data Discovery Tool

Regardless of what type of industry or business you are involved in, the bottom-line goal is to optimize sales; and that involves replacing any archaic tech processes with cutting-edge technology and substituting any existing chaos with results-driven clarity.

Data discovery tools, being a business-intelligence architecture, creates that clarity through the incorporation of a user-driven process that searches for patterns or specific items in a data set via interactive reports. Visualization is a huge component of data discovery tools. One can merge data from multiple sources into a single data set from which one can create interactive, stunning dashboards, reports and analyses. The user is able to observe data come to life via striking visualizations. Furthermore, business users want to perform their own data analysis and reporting—with a data discovery tool they can! After it’s all said and done, smarter business decisions are generated; and that drives results.

Before purchasing a data discovery tool, however, several questions should be addressed:


1: What About Data Prep?
It’s important to realize that there are companies who will claim their data-discovery products are self-service; but keep in mind that many of the products will necessitate a data prep tool in order to access the data to be analyzed. Preparing data is challenging; and if a data prep tool is not included, one must be purchased. Choose a data discovery tool that enables data prep to be handled without any external support.


As a side note: governed self-service discovery provides easy access to data from IT; and an enterprise discovery platform will give IT full visibility to the data and analytic applications while it meets the business’s need for self-service. Business users embrace the independence they are empowered with to upload and combine data on their own.


2: Is Assistance from IT Required for Data Discovery Tools?
Business users desire the ability to prepare their own, personal dashboards and explore data in new ways without needing to heavily rely on IT. Data discovery tools do not require the intervention of IT professionals, yet the relationship with IT remains. Data discovery tools empower the business to self-serve while maintaining IT stewardship. Data discovery tools allow users to directly access data and create dashboards that are contextual to their needs—when they need it and how they need it! This, in turn, reduces the number of requests for reports and dashboards from IT staff and allows those professionals to focus more intently on development projects and system improvements. Software solutions that support data discovery, such as business intelligence platforms with innovative visualization capabilities, are zealously supported by non-technical business users since they can perform deep, intuitive analysis of any enterprise information without reliance on IT assistance.


3: Will Data Discovery Tools Allow One to Interact with the Data?
The fun thing is, you can play with the data to the point of being able to create, modify and drill down on a specific display. A beautiful feature of data discovery tools is the interactive component which permits one to interact with corporate data sources visually to uncover hidden trends and outliers. Data discovery facilitates intuitive, visual-based and multivariate analysis via selecting, zooming, pivoting, and re-sorting to alter visuals for measurement, comparisons and observation.


4: Are Data Discovery Tools Intended for Enterprise Use?
Enabling the business to self-serve while maintaining IT stewardship creates reliable decisions the enterprise can rely on. Data discovery tools are invaluable for enterprise use—organizations can plan their approach to incorporate data discovery tools into their infrastructure and business practice.

Data discovery tools allow one to retrieve and decipher data from spreadsheets, departmental databases, enterprise data warehouse and third-party data sources more efficiently than ever! Multidimensional information can be transformed into striking graphical representations—3D bar and pie charts, histograms, scatter plots and so much more! Data discovery tools deliver enterprise solutions within the realms of business information and analytics, storage, networks & compliance, application development, integration, modernization and database servers and tools.


5: With Data Discovery Tools Can I Retrieve Answers At any Time?
Data discovery tools will allow you to make inquiries and get answers quickly and seamlessly. Geographic location will make no difference since files can be loaded on a laptop or even a mobile phone or other mobile devices. With a few clicks, you can unlock all your data from servers, a mainframe or a PC.

Multiple charts, graphs, maps and other visuals can, all, be combined in analytic dashboards and interactive apps. Answers to crucial questions and issues can be quickly unveiled. Analysts can share the data, with ease, among all users via the web and mobile devices—all operating like a fine-tuned engine—anytime, anywhere.

Data discovery tools are changing business intelligence!


Guest blogger post:
Mike Miranda writes about enterprise software and covers products offered by software companies like rocket software about topics such as Terminal Emulation, Legacy Modernization, Enterprise Search, Big Data, Enterprise Mobility and more.

Why Data Virtualization Is so Vital

In today’s day and age, it probably seems like every week you hear about a new form of software you absolutely have to have. However, as you’re about to find out, data virtualization is actually worth the hype. 

The Old Ways of Doing Things
Traditionally, data management has been a cumbersome process, to say the least. Usually, it means data replication, data management or using intermediary connectors and servers to pull off point-to-point integration. Of course, in some situations, it’s a combination of the three.

Like we just said, though, these methods were never really ideal. Instead, they were just the only options given the complete lack of alternatives available. That’s the main reason you’re seeing these methods less and less. The moment something better came along, companies jumped all over them.

However, their diminishing utility can also be traced to three main factors. These would be:

· High costs related to data movement
· The astronomical growth in data (also referred to as Big Data)
· Customers that expect real-time information

These three elements are probably fairly self-explanatory, but that last one is especially interesting to elaborate on. Customers these days really don’t understand why they can’t get all the information they want exactly when they want it. How could they possibly make sense of that when they can go online and get their hands on practically any data they could ever want thanks to the likes of Google? If you’re trying to explain to them that your company can’t do this, they’re most likely going to have a hard time believing you. Worse, they may believe you, but assume that this is a problem relative to your company and that some competitor won’t have this issue.


Introducing Data Virtualization
It was only a matter of time before this problem was eventually addressed. Obviously, when so many companies are struggling with this kind of challenge, there’s quite the incentive for another one to solve it.

That’s where data virtualization comes into play. Companies that are struggling with having critical information spread out across their entire enterprise in all kinds of formats and locations never have to worry about the hardships of having to get their hands on it. Instead, they can use virtualization platforms to search out what they need.


Flexible Searches for Better Results
It wouldn’t make much sense for this type of software to not have a certain amount of agility built in. After all, that’s sort of its main selling point. The whole reason companies invest in it is because it doesn’t get held back by issues with layout or formatting. Whatever you need, it can find.

Still, for best results, many now offer a single interface that can be used to separate and extract aggregates of data in all kinds of ways. The end result is a flexible search which can be leverage toward all kinds of ends. It’s no longer about being able to find any type of information you need, but finding it in the most efficient and productive way possible.


Keep Your Mainframe
One misconception that some companies have about data virtualization is that it will need certain adjustments to be made to your mainframe before it can truly be effective. This makes sense because, for many platforms, this is definitely the case. These are earlier versions, though, and some that just aren’t of the highest quality.

With really good versions, though, you can basically transform your company’s mainframe into a virtualization platform. Such an approach isn’t just cost-effective. It also makes sure you aren’t wasting resources, including time, addressing the shortcomings of your current mainframe, something no company wants to do.

Don’t get turned off from taking a virtualization approach to your cache of data because you’re imagining a long list of chores that will be necessary for transforming your mainframe. Instead, just be sure you invest in a high-end version that will actually transform your current version into something much better.


A Better Approach to Your Current Mainframe
Let’s look at some further benefits that come from taking this approach. First, if the program you choose comes with the use of a high-performance server, you’ll immediately eliminate the redundancy of integrating from point-to-point. This will definitely give you better performance in terms of manageability. Plus, if you ever want to scale up, this will make it much easier to do so.

Proper data migration is key to a good virtualization process. If it is done right the end user wont have to worry out corrupted data and communication between machines will be crystal clear.

If you divert the data mapping you need to do at processing-intensive level and transformation processes away from the General Purpose Processor of your mainframe to the zIIP specialty engine, you’ll get to dramatically reduce your MIPS capacity usage and, therefore, also reduce your company’s TCO (Total Cost of Ownership).

Lastly, maybe you’d like to exploit of every last piece of value you derive from your mainframe data. If so, good virtualization software will not only make this possible, but do so in a way that will let you dramatically turn all of your non-relational mainframe data virtualization into relational formats that any business analytics or intelligence application can use.


Key Features to Look for in Your Virtualization Platform
If you’re now sold on the idea of investing in a virtualization platform, the next step is getting smart about what to look for. As you can imagine, you won’t have trouble finding a program to buy, but you want to make sure it’s actually going to be worth every penny.

The first would be, simply, the amount of data providers available. You want to be able to address everything from big data to machine data to syslogs, distributed and mainframe. Obviously, this will depend a lot on your current needs, but think about the future too.

Then, there’s the same to think about in terms of data consumers. We’re talking about the cloud, analytics, business intelligence and, of course, the web. Making sure you will be able to stay current for some time is very important. Technology changes quickly and the better your virtualization process is the longer you’ll have before having to upgrade. Look closely at the migration process, and whether or not the provider can utilize your IT team to increase work flow. This will help you company get back on track more quickly and with better results.

Finally, don’t forget to look at costs, especially where scalability is concerned. If you have plans of getting bigger in the future, you don’t want it to take a burdensome investment to do so.

As you can see, virtualization platforms definitely live up to the hype.You just have to be sure you spend your money on the right kind.


Guest blogger post :

Mike Miranda writes about enterprise software and covers products offered by software companies like Rocket software about topics such as Terminal emulation, Enterprise Mobility and more.