Big Data Engineering

Sorted by:  

What Is Middleware?

Loosely defined, middleware is a set of services that perform various functions in a distributed computing environment, across a wide set of server and client systems. In essence, middleware is computer [more…]

Middleware for Data Warehousing

Middleware is computer software that connects software components. In a data warehousing environment, the middleware services are the set of programs and routines that do the following: [more…]

Should You Use Tools or Custom Code?

In the early days of data warehousing, most organizations handled middleware services through custom coding, rather than with the few tools available at the time, as shown in this example: [more…]

Middleware Services: Data Selection and Extractions

The primary purpose of the data-selection and –extraction service is to select from (find in) a data source the data that you want to move into the data warehouse and then [more…]

Middleware Services: Data Quality Assurance

You should establish two different quality assurance (QA) services in the flow of middleware services. You have to perform the first QA tasks against the extract from the data source before you perform [more…]

Middleware Services: Data Mapping and Transformation

This figure shows an environment in which data is being extracted from three different data sources for inclusion in a data warehouse, and each of the three sources is on a different platform. At some [more…]

Middleware Services: Data Movement and Data Loading

In most situations, the two middleware services — selection and extraction, and quality assurance — take place on the same platform (system) on which the data source resides. If your data warehouse will [more…]

Replication Services for Data Warehousing

Replication middleware services combine selection and extraction, movement, and loading from one database to one or more others, usually managed by a single DBMS product. [more…]

What Does Management Need to Know about Data Warehousing?

Somewhere in your organizational hierarchy, somebody has control over budgeted funds that he or she can allocate to your data warehousing project or to another project elsewhere in the organization, to [more…]

How to Convince Your Bosses of the Need for Data Warehousing

When you’re pitching a data warehousing project to top management (even to the CEO or CFO), work the following two statements into your presentation: [more…]

Data Warehousing and Business Trends

Now that the data warehousing era is here, the next generation of business and management trends (you had better believe that a next generation will come along) might have a little more substance — a little [more…]

Data Warehousing in a Cross-Company Setting

Data warehousing is usually a private affair. Even when external data about your competitors is part of your environment, it’s still your company’s data warehouse, built for your company’s benefit and [more…]

Executive Leadership and Data Warehousing

Executive management doesn’t care one iota about data warehousing. They don’t. It’s that simple. They do care about the business value a successful data warehousing project delivers. [more…]

Data Warehousing and the Infrastructure Challenge

The nature of a data warehouse (that it’s composed primarily, or exclusively, of data that comes from elsewhere, other application databases, and is converted into a data asset) means that it can’t stand [more…]

Data Warehouse: Source Data Feeds

A data warehouse is, by its very nature, a distributed physical data store. Distribution of your information assets assists in the performance and usability across systems and across the enterprise. Make [more…]

Data Warehousing: Operational Data Store (ODS)

Some definitions of an ODS make it sound like a classical data warehouse, with periodic (batch) inputs from various operational sources into the ODS, except that the new inputs overwrite existing data. [more…]

Data Warehousing: The ODS Defined

Here’s a working definition of an ODS (it’s a long one): an informational and analytical environment that reflects at any point the current operational state of its subject matter, even though data that [more…]

Data Warehousing: An ODS Example

Here is a data warehousing example to help you understand ODS (operational data store). Suppose that you work in a large financial company that provides a variety of services to elite companies and individuals [more…]

Master Data Management (MDM)

In recent years, ODS-style feedback systems defined for a specific purpose — reference data — have emerged. All systems are packed with reference data. This data can include the set of data you use to [more…]

Service-Oriented Architecture (SOA)

If you establish more data integration by using ODS and MDM data stores, you also need a messaging, or communication, architecture to enable systems that weren’t built to communicate with each other to [more…]

Ensure the Quality of Incoming External Data

When you are designing a data warehouse and you determine what external data you need, you just place an order (similar to ordering clothes or a fruit basket from an online site). After you begin receiving [more…]

Determine What External Data You Really Need

Don’t overdo it when you think about what external data you need for your data warehouse. The same rule that applies to internal data in your warehouse is just as applicable to externally sourced data: [more…]

Why Is External Data Important to Your Data Warehouse?

External data — from outside your own company — is important to your data warehouse for one simple reason: To ensure that you make the right business decisions, you need to see the big picture, which usually [more…]

Identify Data You Need for Your Data Warehouse

Simply ask yourself this question: Looking at a complete list from within your company of all data sources that will provide data to the warehouse, what else do you need that you don’t already have on [more…]

View External Data from a User’s Perspective

Here is an outline of how a data warehouse user sees external data and its importance. Consider these two tables. This table shows the sales performance for Good Guys, Inc., and the following table shows [more…]


Sign Up for RSS Feeds

Computers & Software