Query Clause Indexing

Are the World Governments (Politicians) not spending money and borrowing money because of Global Indian Computing Standards?

Imagine being able to index any CLAUSE of the SQL Query...

I am not able to patent this as I have no Time and Money and on top of that no reply from Indian Patent Officers. 

The Query Clause Indexing can UNITE the WORLD for a Cause. 

Real-Time ETL

Blockbuster of Sorts in the making by INDIA...

FIRST MOVERS ADVANTAGE TOWARDS THE KOLAGATLA'S CAN CONTINUE TO SELL AND BUY IN THE USA DOLLARS PER SE...

SEMI-JOIN A Beautifully Architected Algorithm by TIBCO.

 An algorithm that is so amazing and felt like a hack in Data Virtualization.

A query in the clinical trials data of Astra Zeneca was taking around 2 hours to execute and a lot of caching errors.

It was called the clinical trials hub and I was tasked to optimize it. Semi-Join, Multi Table cache refresh along with drop and recreate indexes, the execution time of the refresh was 2 Minutes.

20 seconds - Execution of Query.
The rest of time - Insertion.

I was amazed because it had been long time since I got my head to work again but something kept running in my mind.

What if I would like to build something superior than the semi-join?

I would have to keep thinking for ever.

When there are two tables that are to be joined from Two Different Databases (Different DB Servers), SEMI-JOIN will convert the parent query in such a way the IN CLAUSE is re-written and the FETCH happens from a Single DataBase Server. I do not need to explain more as a person familiar with SQL will understand how this JOIN Algorithm works. 

Connecting to HIVE (v0.9) from CIS v6.2.6

Hi,

On popular demand I am writing this post on how to connect to HIVE from Cisco Information Server.

We are going to see how to connect to HIVE v0.9 pre-configured on the cloudera vm (v4.21). I know it is an old version but it does the job. I assume you have gone through the articles in this blog and you have an initial understanding of the CIS workflow (Introspect, Model, Publish).

Also, creation of a database/table and insertion of data into the HIVE tables is not a part of this post.

I have created a database called "realty" and also created a table called "listings". Loaded data into the listings table with 6 records. Issued the select * from listings table from the HIVE prompt and could retrieve back the results. Screenshot below:

Now the next part is to start the HIVE thrift server by issuing the following command from the shell prompt:

NOTE: After the Hive Thrift Server is started you should not close the shell window on which the Thrift Server was started. When you interact with the Hive server from CIS you will get messages from the prompt about the ongoing events (Clients to Hive Thrift Server).

Now we can move to CIS to configure the data source and test the querying.

Create a datasource in CIS, selecting the (Hive 0.9.0) as the datasource adapter:

Click Next and provide the server IP Address of the Cloudera VM and give the port as 10000. Change the database name to your HIVE Metastore database (realty in my case) and hive Create and Introspect.

We can see the listings table and can add it to the CIS workspace.

Now the final step is to query the listings table in CIS:
You can do a right click on listings and hit show contents:
it will display the 6 rows in the table as below:

Now this table can be queried and views/web services can be created on top of this introspected listings table.

NOTE: If you get this error message while selecting the HIVE datasource in CIS: Time to approach the administration group of the CIS server.

The admin will configure the HIVE jdbc adapter by copying the adapter files from hadoop to the CIS server.

Connecting to a Public REST API in CIS

Lets see how to connect to a Public REST API in CIS.

The REST API I have selected is the Google Maps API.
URL: http://maps.googleapis.com/maps/api/geocode/xml?address={input_address}
input_address is the input you give to the URL at run time. For Eg: Any valid address like (San Diego)

This URL has two parts the BASE URL and the OPERATION URL

From the Google Maps URL:
BASE URL: maps.googleapis.com
OPERATION URL: /maps/api/geocode/xml?address={input_address}

1.  Login into CIS and click on a folder where you have write access and select New Data Source:

2. It will bring a pop-up where you can select the Data Source Adapter:
Select REST as your Data Source Adapter:

3. Click on Next at the bottom of the screen and you can enter the connection information as below.

Name: Any Valid Name. I have named mine as GOOGLE_REST_API.
Base URL: http://maps.googleapis.com
No Need to enter any Login/Password as anyone can access these Public REST API's .

4. Now in the Operations Tab Click on the + Symbol to add a new operation.

5. A window will pop-up where you enter any name for your operation. Give it any name.

6. Now enter the Operation URL in the Operation URL Text Box:
Our Operation URL: /maps/api/geocode/xml?address={input_address} and Hit Parse.
When you hit Parse it will Parse the Operation URL and detect the input parameters in the URL.

7. Hit Create and Close in the Bottom End of the Window and it will Introspect the Operation by combining the Base URL and The Operation URL.

Click on the Operation Name and Click Next

8. Hit Finish and It will introspect and give you the Introspection Task Status Report. Make sure the operation is added in the report as shown below.

9. Click OK and you can see your operation in the resource directory. Open the operation and click on Execute.

10. You have to enter a valid input_address so the Google Maps API can give you the appropriate information like Country Information, Latitude, Longitude Information etc. The Information is in XML so you can use this XML in SQL Scripts, Views and publish them for the consuming applications.

Note: If you would like to convert XML to tabular format you will have to use the XSLT Transformations provided by CIS.