The Internet Query Engine^TM Architecture

Internet Query Engine: a 3-Layer Architecture

In order to be scalable, flexible and extensible, InterQuery chose a 3-layer architecture for its IQE:

• At the top layer of the model, the application interfaces provide programmers the tools they need to integrate the IQE into their Application Servers, Enterprise Portal Servers, or browser-based systems.

• At the bottom layer of the model, the data connectors provide connectivity to the data sources.

In the middle layer, the heart of the Internet Query Engine, is the Internet Request Engine and Query Engine, responsible for mapping and formatting the user query to the individual format of each data source, and the Parse Engine and Join Engine responsible for transforming and joining the resulting data, and presenting the user a unified data set.

Figure 1: Internet Query Engine - The 3-Layer Architecture

Client Application Integration: the Freedom of Choice

Application developers have the choice among five leading methods to utilize the IQE, all based on industry standards, thus allowing them to focus on their main task in the environment of their choice:

• J2EE is provided as a set of Enterprise Java Beans (EJBs), and can be used in conjunction with any J2EE-compliant Application Server.
• SOAP is accommodated in the IQE and is provided a set of Web Services accessible from any SOAP/UDDI compliant Enterprise Portal. The IQE can be installed as a stand-alone set of Java classes (i.e. without requiring a J2EE application server) that can be registered as web services in an UDDI-compliant directory.
• ODBC allows programmers to treat data sources as a traditional database, and use SQL queries to access data.
• XML allows programmers to use HTTP to send queries, and receive the results as an XML document.
• HTML permits application developers to simply create web forms to query data, and format HTML result pages.

Data Connectors: Reaching Data Anywhere, Anytime

Data connectors are "plug-and-play" components that provide access to data sources. InterQuery has developed and licensed many data source connectors, and also provides the ability to add new connectors. Each data source is accessed via its native interface, and the role of the connector is to map the user request from the client application to the format required and expected from the particular data source. Connectors can operate in three different modes: synchronous, asynchronous, and event-driven. The behavior of connector is partly driven by the External Query Language programs, which contain data manipulation instructions performed by the connector, before the data is passed to the Data Joiner.

Query and Join Engines: Where the Work is Done

The Query Engine and Join Engines are the core of the Internet Query Engine. The role of the Query Broker is to propagate the user query to all of the data sources that are supposed to participate in the request, by submitting individual requests to the data connectors. In turn, the Data Joiner receives the responses from the various connectors, and assembles the data into a single unified data set.

Figure 2: Internet Query Engine - Query Engine and Join Engine

Query Engine

The Query Engine has several major roles: thread pooling, security, and translation into the abstraction layer format.

• Thread Pooling: Through the metadata repository, and based on which data view the query is requesting the Query Engine establishes the list of data sources to be queried. The Query Engine then creates a pool of threads for use by the data source Connectors. The number of threads allocated to a query depends on the policy set by the system administrator, and on the current workload. A throttling algorithm also allows the requesting application to decrease the number of threads allocated to a single query as the load of the system increases. In addition, to avoid overloading data sources with too many requests, the Query Engine can limit the number of simultaneous connections to each data source.
• Security: The Query Engine invokes the various connectors, after applying access right policies contained in the metadata repository.
• Translation: The Query Engine translates the query into the internal common representation independent of the data source (the Abstraction Layer, see InterQuery Concepts section).

Join Engine

The Join engine receives the results from the various connectors, and assembles (joins or unions) the data into a single result set. It then applies the data manipulation rules and the business rules that are relevant to the entire result set (the data manipulation rules relevant to each individual data source are processed by the connectors). The Join Engine is also responsible for the streaming and caching functions.

External Query Language Programs

For every view created by the application designer, every data source is represented in the IQE via one or more External Query Language Program (an EQL). The number of EQLs is specific to each data source and depends on the richness of the data source interface. A data source with many methods and data sets (e.g. Oracle database) can be represented by many EQLs. InterQuery provides a number of pre-defined IQLs in its product suite. Additional EQLs may be created by users to match specific application functionality.

Base Views

Base Views are comprised of a View definition and a single EQL. These EQLs are "data source facing", meaning that each EQL contains the necessary logic, and connector information, transformation and mapping details to bring the data source into the engine. Base views, as a group, constitute the system's meta-data repository, providing the necessary "application facing" information to facilitate the creation of Application views.

Application Views

Application Views are comprised of View definitions and one or more EQLs which utilize the Base Views to create relationships across the underlying data sources in the system. These relationships - joins and unions - are instantly available to the Client Application as soon as the Application View has been saved. Like Base Views, Application Views include transformation and mapping details, as well as optional presentation information.

Figure 3: Putting it all Together

Conclusion

Rapid Deployment, High Performance, Flexibility, and Ease-of-Use

InterQuery's Internet Query Engine was build with rapid deployment, performance, flexibility and ease-of-use as guiding principles. Its open, standards-based and flexible architecture is in itself a guarantee that it will continue to evolve, deliver more value, and accommodate rapidly changing information delivery needs and environments.