bsc thesis
TRANSCRIPT
MULTI KEY INDEXING FOR DISTRIBUTED DATABASE MANAGEMENT SYSTEM
BY
MD. SHAZZAD HOSAIN STUDENT NO: 9505025
For the partial fulfillment of B.Sc. Engineering Degree in Computer Science and Engineering
SUPERVISED BY MD. HUMAYUN KABIR
Assistant Professor &
ABDUL HAKIM NEWTON Lecturer
Department of Computer Science & Engineering, BUET
Department of Computer Science & Engineering Bangladesh University of Engineering & Technology, Dhaka – 1000, Bangladesh
Acknowledgement
I like to express my sincerest appreciation and profound gratitude to my supervisors
Md. Humayun Kabir, Assistant Professor and Abdul Hakim Newton, Lecturer,
Department of Computer Science and Engineering, BUET, for their supervision,
encouragement and guidance. Especially, Md. Humayun Kabir has keen interest in
distributed database, and his valuable suggestions and advice were the source of all
inspirations to me. I also like to convey gratitude to all my course teachers here. Their
teaching helps me a lot to start and complete this thesis work. Finally, I would like to
acknowledge the assistance and contribution of a large number of individuals and
express my gratefulness to them.
(Md. Shazzad Hosain )
Abstract Complex accessing structures like indexes are a major aspect of centralized database.
The support for these structures is the most important part of the database
management systems (DDMS). The reason for providing indexes is to obtain fast and
efficient access to data. Most of the centralized database management systems use B-
tree or other types of index structures such as bit vector, graph structure or grid file
index. But in distributed databases no index structure is used to obtain fast and
efficient access to the data. Therefore efficient access is the major problem in
distributed databases. We proposed a distributed index model, which is a data
structure based index comprising of two types of index structures: Global Index (GI)
and Local Index (LI). GI is created and maintained by distributed database component
(DDB) and LI is created and maintained by local database component (DB) of a
distributed database management system. Our proposed global index (GI) uses the
techniques of bit vector, graph structure and grid file organization. This distributed
database index is implemented with multi key indexing technique, which enables us
to search a record with more than one attribute values. A simulation program tested
the proposed model and found satisfactory results.
Chapter 1
Introduction
1.1 An overview of distributed database system
The traditional database approach keeps all data centrally and then accesses them
mostly in a client server model. But in a distributed database system data are
distributed over site geographically. Let us clarify this feature through an example.
Say there are three branch of a bank at different site. There will be two types of
transaction, one is local transaction and the other is global transaction. Local
transaction means money transaction in the account at the same site. But if it needs to
transfer money from one account of a site to another account of another site then a
global transaction will occur. In that case the program has to access data over site,
which needs much attention, such as, transaction over network, speed, efficient
access, integrity, recovery, concurrency control, privacy, security and a lot of things.
These entire tasks are done by DDBMSs (Distributed Database Management System).
When a local operation happens all are right as usual, but when a global operation
needs it is the DDBMSs to determine which site has to access or how to perform the
operation.
An important property of DDBMSs is whether they are homogeneous or
heterogeneous. Homogeneity and heterogeneity can be considered at different levels
in a distributed database, the hardware, the operating system and the local DBMSs.
However, the important thing for us is at the level of local DBMSs, because the
communication software manages differences at lower levels. Therefore the term
homogeneous DDBMSs, refers to a DDBMSs with the same DBMS at each site, even
if the computers and / or the operating system are not the same. A heterogeneous
DDBMS uses instead at least two different DBMSs. This adds the problem of
translating between the different data models of the different local DBMSs to the
complexity of homogeneous DDBMSs. So if Top-Down design of a global schema
for a new system has to be designed a homogeneous system is a fine solution but if
there already exists different local DBMSs and one has to integrate those systems then
obviously heterogeneous system will emerge and in that case the DDBMSs has to
cope with this heterogeneous system.
1.2 Reference architecture of distributed database
In order to understand distributed database system we have to study this Reference
architecture of distributed database architecture. Actually it is not explicitly
implemented an all distributed databases but we have to analyze and understand all
the components of this architecture to have better knowledge about distributed
database.
Fig 1.1: The reference architecture for distributed database.[8]
The reference model has two main parts
1. Site independent schemas
2. Site dependent schemas
Global Schema
Fragmentation Schema
Local mapping Schema 1
Local mapping Schema 2
DBMS of site 1 DBMS of site 1
Local database At site 1
Local database At site 2
(Other site)
Allocation schema
1.2.1 Site independent schemas:
This schemas has the following parts
• Global schema
• Fragmentation schema
• Allocation schema
1.2.1.1 Global schema:
This schema defines all the data that are contained in the distributed database as if
the database were not distributed at all. Therefore, this schema is defined exactly as
the schema of no distributed database. However, the data model that is used for the
definition of a global schema should be compatible for the definition of the mapping
to the other levels of the distributed database. For this purpose, the relational data
model will be used. Using this model, the global schema consists of the definition of a
set of global relations.
1.2.1.2 Fragmentation schema:
Each global relation can be split into several no overlapping portions that are
called fragments. The mapping between global relations and fragments is defined as
fragmentation schema. This mapping is one to many: i.e., several fragments
correspond to one global relation, but only one global relation corresponds to one
fragment. Fragments are indicated by a global relation name with an index (fragment
index); for example, Ri indicates the ith fragment of global relation R.
1.2.1.3 Allocation schema:
Fragments are logical portions of global relations that are physically located at one
or several sites of the network. The allocation schema defines at which sites a
fragment is located. All the fragments that correspond to the same global relation R
and are located at the same site j constitute the physical image of global relation at site
j. Therefore there is a one to one mapping between a physical image and a pair; a
global relation named and a site index can indicate physical images. To distinguish
them from fragments, we will use a superscript; for example, Rj indicates the physical
image of the global relation R at site j.
An example of the relationship between the object types defined above is shown
in fig 2. A global relation R is split into four fragments R1, R2, R3, R4. These four
fragments are allocated redundantly at the three sites of a computer network, thus
building three physical images R1, R2 and R3.
R
Global relation Fragments Physical image
1. FIG 1.2: FRAGMENTS AND PHYSICAL IMAGES FOR A GLOBAL RELATION [8]
To complete the terminology, we will refer to a copy of a fragment at a given site,
and denote it using the global relation name and two indexes. For example, in fig 2,
the notation R32 indicates the copy of fragment R2 that is located at site 3.
Finally, note that two physical images can be identical. In this case, we will say
that a physical image is a copy of another physical image. For example, in fig 2 R1 is
a copy of R2.
1.2.2 Site dependent schemas:
This schemas has the following parts
• Local mapping schema
• DBMS of the local site
• Local database at that site
R1
R2
R3
R4
R1 site 1
R2 site 2
R3 site 3
R11
R12
R21
R22
R31
R32
R33
1.2.2.1 Local mapping schema:
Since the top three levels are site independent, therefore they do not depend on the
data model of local DBMS. At a lower level, it is necessary to map the physical
images to the objects that are manipulated by the local DBMS. This mapping is called
a local mapping schema and depends on the type of local DBMS; therefore, in a
heterogeneous system we have different types of local mappings at different sites.
This reference architecture provides a very general conceptual framework for
understanding distributed database system. The most important three features that
motivates in designing this architecture are
• Separation of data fragmentation and allocation.
• The control of redundancy.
• The independence from local DBMS.
• Separation of data fragmentation and allocation:
This separation allows us to distinguish two different levels of distribution
transparency, namely fragmentation transparency and location transparency.
Fragmentation transparency is the highest degree of transparency where as the
location transparency is the lower degree of transparency. The separation between the
concept of fragmentation and allocation is very convenient for the distributed
database because the determination of relevant portions of the data is thus
distinguished from the problem of optimal allocation.
• Explicit control of redundancy:
In fig 2 the two physical images R2 and R3 are overlapping; i.e. they contain
common data. The definition of disjoint fragments as building blocks of physical
images allows us to refer explicitly to this overlapping part: the replicated fragment
R2. As we shall see, the explicit control over redundancy is useful in several aspects
of distributed database management.
• Independence from local DBMS:
The feature local mapping transparency allows us to build distributed database
system homogeneous or heterogeneous. In a homogeneous system, it is possible that
the site independent schemata are defined using the same data model as the local
DBMS but in a heterogeneous database system local mapping schemata helps to
coordinate the different kinds of DBMS.
Another kind of transparency that is strictly related to location transparency is
replication transparency. Replication transparency means that the user is unaware of
the replication of fragments.
1.2.3 Types of data fragmentation:
The decomposition of data fragmentation is two types, horizontal fragmentation
and vertical fragmentation. More fragmentation that is complex can be obtained by
combining these two types of fragmentation. In all types of fragmentation, a fragment
can be defined by an expression in a relational language that takes global relations as
operands and produces the fragment as result. There are some rules that must be
followed when defining fragments.
1.2.3.1 Completeness condition:
All the data of the global relation must be mapped into the fragments; i.e., it must
not happen that a data item that belongs to global relation does not belong to any
fragment.
1.2.3.2 Reconstruction condition:
It must be always possible to reconstruct the global relation from those fragments.
This necessity must comply with this architecture because distributed database only
store the fragments in different sites and global relation have to be built through this
reconstruction operation if necessary.
1.2.3.3 Disjointness condition:
It is convenient that fragments should be disjoint, so that the replication of data
can be controlled explicitly at the allocation level. However, this condition is satisfied
by horizontal fragmentation while the in vertical fragmentation this condition
sometimes violated.
1.2.4 Horizontal fragmentation:
Horizontal fragmentation consists of partitioning the tupelos of a global relation
into subsets that is very much useful for the distributed database system. If data are
fragmented by some common properties then those data can be stored geographically
in a convenient way. Here we can clarify this by an example. Let a global relation be
SUPPLIER (SUM, NAME, CITY)
Here the SUPPLIER contains supplier number, supplier name and the city where
the supplier lives. However if the entire supplier comes from Sanfransisco city (SF”)
and Losangels city (“LA”) then the horizontal fragmentation can be defined in the
following way:
SUPPLIER1 = SL CITY = ”SF” SUPPLIER
SUPPLIER2 = SL CITY = ”LA” SUPPLIER
The above fragmentation satisfies the completeness condition because “SF” and
“LA” are the only possible values of CITY attribute, otherwise we would not know to
which fragments the other CITY values belong.
Again, the reconstruction condition is easily verified, because it is always possible
to reconstruct the SUPPLIER global relation through the union operation:
SUPPLIER = SUPPLIER1 UN SUPPLIER2
1.2.5 Vertical fragmentation:
The vertical fragmentation can be obtained by the subdivision of its attributes into
global groups. It is useful when the subgroups have the same common geographical
properties. For example a global relation
EMPLOYEE (EMPNUM, SAL, TAX, MGRNUM, DEPTNUM)
A vertical fragmentation of this relation can be defined as
EMPLOYEE1 = PJ EMPNUM, NAME, MGRNUM, DEPTNUM EMPLOYEE
EMPLOYEE2 = PJ EMPNUM, SAL, TAX EMPLOYEE
The fragmentation could, for instance, reflect an organization in which salaries
and taxes are managed separately. The reconstruction of relation EMPLOYEE can be
obtained as
EMPLOYEE = EMPLOYEE1 JOIN EMPNUM = EMPNUM EMPLOYEE2
1.2.6 Mixed fragmentation:
The fragments that are obtained by the above fragmentation operations are
recursively. The global relation finally can be obtained by the reconstruction rules in
inverse order; consider for example, the same global relation
EMPLOYEE (EMPNUM, NAME, SAL, TAX, MGRNUM, DEPTNUM)
The following is a mixed fragmentation that is obtained by applying the vertical
fragmentation of the previous example, followed by a horizontal fragmentation on
DEPTNUM:
EMPLOYEE1 = SL DEPTNUM <= 10 PJ EMPNUM, NAME, MGRNUM, DEPTNUM EMPLOYEE
EMPLOYEE2 = SL 10 < DEPTNUM <= 20 PJ EMPNUM, NAME, MGRNUM, DEPTNUM EMPLOYEE
EMPLOYEE3 = SL DEPTNUM > 10 PJ EMPNUM, NAME, MGRNUM, DEPTNUM EMPLOYEE
EMPLOYEE4 = PJ EMPNUM, NAME, SAL, TAX EMPLOYEE
The reconstruction of relation EMPLOYEE is defined by the following expression:
EMPLOYEE = UN (EMPLOYEE1, EMPLOYEE2, EMPLOYEE3) JN EMPNUM=EMPNUM
PJ EMPNUM, SAL, TAX EMPLOYEE4
Here the convention followed
JN = JOIN UN = UNION SL = SELECTION PJ = PROJECTION
EMPLOYEE
EMPLOYEE4
2. FIG 1.3: THE FRAGMENTATION TREE OF RELATION EMPLOYEE[8]
Mixed fragmentation can be conviniently presented by a fragmentation tree. In a
fragmentation tree, the root corresponds to a global realtion, the leaves correspind to
the fragments, and the intermediate nodes results of the fragment-defining
expressions.
H
EMPLOYEE1 EMPLOYEE1 EMPLOYEE1
V
Chapter 2
Design of distributed database system
2.1 Objectives of the design of data distribution:
In the design of data distribution we have to consider the following objectives
2.1.1 Process locality:
In a distributed database system there must be remote refference as well as local
reference. Since remote reference costs more than local reference it is the main issue
to minimize the remote reference. Thus in distributing data to minimize remote
reference or to maxize local reference we have to place data as close as possible to the
application which use them.
Designing data distribution for maximizing procesing loaclity can be achieved by
adding the number of local and remote references corresponding to each candidate
fragmentation and fragment allocation and selecting the best solution among them.
2.1.2 Availability and reliability of distributed data:
A high degree of availability for read only aplications is achieved by storing
multiple copies of the same information. The system must be able to switch to an
alternative copy when the one that should be accessed under normal condition is not
available.
Reliability can also be achieved by storing multiple copies of data. If one copy is
destroyed by events which have destroyed computers or storage, data can be available
from other copy of data. Since physical destruction can be caused by fire, earthquake
or sabotage, it is relevant to store replicated copies in geographically dispersed
locations.
2.1.3 Workload distribution:
Distributing the workload over the sites is an important feature of distributed
computer system. Workload distribution is done in order to take advantage of the
different powers or utilizations of computers at each site, and to maximize the degree
of parallelism of execution of applications.
2.1.4 Storage cost and availability:
Though the cost of data storage is not relevant if compared with CPU, I/O, and
transmission cost of applications, but the limitation of available storeage at each site
must be considered.
2.2 Distributed database design:
There are two attarctive approaches to design the destributed databas system. One
is top-down and the other is bottom-up approach.
In the top-down approache, we start by designing the global schema, and we
proceed by designing the fragmentation of the database, and then allocationg the
fragments to the sites, creationg the physical images. This approach is the most
attractive for systems which are developed from scratch, since it alows performing the
design rationally.
While top-down appreach is attractive for the system which is to be installed for
the first time. But for an existing system this approach is not prefferd. In that case
existing databases are aggregated, a bottom-up approach. This approach is based on
the integration of existing schemata into a single, global schema. By integrating it we
mean the merging of commonn data definitions and the resolution of conflicts among
different representations given to the same data. In summary, the bottom-up approach
of a distributed database system rewuires:
1. The seelction of a common database model for describing the global schema of
the database.
2. The tarnslation of each local schema into the common data model.
3. The integration of the local schemata into a common global schema.
2.2.1 The allocation of fragments:
The data allocation problem has been widely analyzed in the context of the ‘file
allocation problem’. The easiest way to apply this work to the fragment allocation
problem is to consider each fragment as a separate file; however, this approach is not
convinient for the following reasons:
1. Fragments are not properly modeled as individual files, since in this way we do
not take into account the fact that they have the same structure or behavior.
2. There are many more fragmnets then original global relations, and many analytic
models cannot compute the solution of problems involving too many variables.
3. Modeling application behavior in file systems is very simple, while in distributed
database applications can make a sophisticated use of data.
Some of this problems have not yet been solved satisfactorily; for instance,
problem 3 is particularly difficult, since the correct approach would be to evaluate
data distribution measuring how optimized aplications should behave with it. This,
however, requires oprimizing all the important applications for each possible data
allocation; solving query optimization as a subproblem of data allocation is probably
too hard, since it requires either an excessive amount of simulation or an excessively
complex analytic computation.
2.2.2 General criteria for fragment allocation:
In determining the allocation of fragments, it is important to distinguish whether
we design a final nonredundant or redundant allocation. The simplest method for a
nonredundant final allocation is a best fit approach; a measure is associated with each
possible allocation. However replication introduces further complexity in the design,
because:
1. The degree of replication of each fragment becomes a variable of the problem.
2. Modeling read applications is complicated by the fact theat the applications can
now select among several alternative sites for accessing fragments.
For determining the redundant alloation of fragments, either of the following
methods can be used:
1. All beneficial sites: In this approach the set of all sites where the benefit of
allocation one copy of the fragment is higher than the cost, and allocate a copy of
the fragment to eache element of this set.
2. Additional replication: Here first the solution of the nonreplicated problem, and
then progresively introduce replicated copies starting from the most benefiicial;
the process is terminated when no additional replication is beneficial.
2.2.3 Measure of costs and benefits of fragment allcation:
Here we wile give some very simple formulas for evaluating costs and benefits of
the allocation of fragments of a global relation R. let us first introduce some
definitions:
• i is the fragment index
• j is the site index
• k is the application index
• fkj is the frequency of application k at site j
• rki is the number of retrival references of application k to fragment i
• uki is the number of update references of application k to fragment i
• nki = rki - uki
2.2.3.1 Horizontal fragmentation:
1. Using the ‘best-fit’ approach for a nonreplicated allocation, we place Ri at the site
where the number of references to Ri is maximum. The number of local references
of Ri at site j is
Bij = ∑k fkj nki [8]
Ri is allocated at site j* such that Bij* is maximum.
2. Using the ‘all beneficial sites’ method for replicated allocation, we place Ri at all
sites j where the cost of retrieval references of applications is larger than the cost
of update references to Ri from applications at any other site. Bij is evaluated as
the diference:
Bij = ∑k fkj rki – C * ∑k∑j’≠j fkj’ uki [8]
C is a constant which measures the ratio between the cost of an update and a
retrieval access; typically, update accesses are more expensive, since they rewuire
a larger number of control messages and local opereations.
3. Using the ‘additional replication’ method for replicated allocation, we can
measure the benefit of placing a new copy of Ri in terms of increased reliability
and availability of the system. The benefit does not grow proportionally to the
degree of redundancy of Ri. Let di denote the degree of redundancy of Ri, and let
Fi denote the benefit of having Ri fully replicated at each site. Here the following
function β(di) was introduced to measure this benefit:
β(di) = (1 – 21-di) FI [8]
Note that β(1) = 0, β(2) = Fi / 2, β(3) = 3 Fi / 4, and so on. Then we evaluate the
benefit of introducing a new copy of Ri at site j by modifying the formula of case
2 as follows:
Bij = ∑k fki rki – C * ∑k ∑j’ ≠ j fkj / uki + β(di) [8]
The formula takes into account the degree or replication.
Chapter 3
Query optimization
3.1 Translation of global queries to fragment queries
So far we have seen that an access operation issued by an application can be
expressed as a query, which references global relations. The DDBMS has to transform
this query into simpler queries, which refer only to fragments. In general, there are
several different ways to transform a query over global relations (called global query)
into queries over fragments (called fragment queries). These different transformations
produce fragment queries, which are equivalent, in the sense that they produce the
same result and this is known as equivalent transformation.
3.1.1 Equivalence transformations for queries:
A relational query can be expressed using different languages; however here we
will use relational algebra and SQL for this purpose because it is possible to transform
most SQL queries into equivalent expressions of relational algebra and vice versa. We
can interpret an expression of relational algebra not only as the specification of the
semantics of a query but also as the specification of a sequence of operations. From
this viewpoint, two expressions with the same semantics can describe two different
sequences of operations. For example,
PJ NAME, DEPTNUM SL DEPTNUM = 15 EMP
and
SL DEPTNUM = 15 PJ NAME, DEPTNUM EMP
are equivalent expressions but define two different sequences of operations.
3.1.2 Operator tree of a query:
In order to have a more practical representation of queries, in which expression
manipulation is easier to follow, operator tree is introduced. Let us consider query Q1,
which requires the supplier number of suppliers that have issued a supply order in the
North area of our company. Query Q1 corresponds to the following expression of the
relational algebra:
Q1 : PJ SNUM SL AREA = “North” (SUPPLY JN DEPTNM = DEPTNUM DEPT)
The operator tree of the corresponding query is given bellow:
PJ SNUM
SL AREA = “North”
JN DEPTNUM = DEPTNUM
SUPPLY DEPT
Fig 3.1 : An operator tree for query Q1.[8]
3.1.3 Equivalent transformations for the relational algebra:
Two relations are equivalent when their tuples represent the same mapping from
attribute names to values, even if the order of attribute is different, equivalent
transformations can be given systematically for small expressions, i.e., expressions of
two or three operand relations. These transformations are classified into categories
according to the type of the operators involved. Let U and B denote unary and binary
algebraic operations, respectively. We have:
• Commutativity of unary operations
U1U2R ↔ U2U1R
• Commutativity of operands of binary operations:
RBS ↔ SBR
• Associativity of binary operations:
RB (S B T) ↔ (R B S) BT
• Idempotence of unary operations:
UR ↔ U1U2R
• Distributivity of unary operations with respect to binary operations:
U (R B S) → U (R) B U (S)
• Factorization of unary operations
U (R) B U (S) → U (R B S)
In nondistributed databases, general criteria have been given for applying
equivalence transformations for the purpose of simplifying the execution of queries:
Criterion 1. Use idempotence of selection and projection to generate appropriate
selections and projections for each operand relation.
Criterion 2. Push selections and projections down in the tree as far as possible.
These criteria descend from the consideration that binary operations, and
specifically joins, are the most expensive operations of database systems, and
therefore it is convenient to reduce the sizes of operands of binary operations before
performing them. In distributed databases, these criteria are even more important:
binary operations require the comparison of operands that could be allocated at
different sites. Transmission of data is one of the major components of the costs and
delays associated with query execution. Thus, reducing the size of operands of binary
operations is a major concern.
PJ SNUM
JN DEPTNUM = DEPTNUM
PJ NAME.DEPTNUM PJ DEPTNUM
SUPPLY SL AREA = “NORTH”
DEPT
Fig 3.2: A modified operator tree for query Q1[8]
Fig2 shows a modified operator tree for query Q1, in which the following
transformations have been applied:
1. The selection is distributed with respect to the join; thus, the selection is
applied directly to the DEPT relation.
2. Two new projection operations are generated and are distributed with respect
to the join.
3.1.4 Operator graph and determination of common sub expressions
An important issue in applying transformations to a query expression is to
discover its common sub expressions’ i.e., sub expressions which appear more than
once in the query; clearly, there is a saving if common sub expressions are evaluated
only once. Here an example is given to illustrate this method. Let Q2: give the names
of employees who work in a department whose manager has number 373 but who do
not earn more than $35,000. An example for this query is:
Q2: PJ EMP.NAME (( EMP JN DEPTNUM = DEPTNUM SL MGRNUM = 373 DEPT ) DF ( SL SAL >
35000 EMP JN DEPTNUM = DEPTNUM SL MGRNUM = 373 DEPT ))
The corresponding operator tree and the operator trees after improving is given in the
next page:
3.2 TRANSFORMING GLOBAL QUERIES INTO FRAGMENT
QUERIES
In this section we give a standard transformation, which maps an algebraic
expression over the global schema into an algebraic expression over the fragmentation
schema.
3.2.1 Canonical expression of a fragment query
Given an algebraic expression over the global schema, its canonical expression is
obtained by substituting, for each global relation name appearing in it, the algebraic
expression giving the reconstruction of global relations from fragments. In the same
way, we map an operator tree on the global schema to an operator tree on the
fragmentation schema by substituting for the leaves of the first tree the corresponding
expressions of the inverse of the fragmentation schema. The important fact is that the
leaves of the operator tree of the canonical expression are now fragments rather than
global relations.
Fig4 (A) shows the transformation of the operator tree of query Q1 represented in
fig2 into the operator tree of the canonical expression of Q1. In fig4(A) the two
circled sub trees substitute the global relations SUPPLY and DEPT of fig2.
3.2.2 Simplifications of Horizontally Fragmented Relations
Let us consider, for simplicity, the join between two fragmented relations R and S.
There are two distinct possibilities of joining them; the first one requires collecting all
the fragments of R and S before performing the join. The second one consists of
performing the join between fragments and then collecting all the results into the
same result relating we refer to this second case as “distributed join”. Neither of the
above possibilities dominates the other. Very generally, we prefer the first solution if
conditions on fragments are highly selective; the second solution is preferred if the
join between fragments involves few pairs of fragments. Join graph help the design
the fragments that need to be joined together. Building the join graph and then using
the following criteria simplification is made:
Criteria 1: In order to distributed joins which appear in the global query, unions
(representing fragment collections) must be pushed up, beyond the joins that we want
to distribute.
Criteria 2: Use the algebra of qualified relations to evaluate the qualification of
operands of joins; substitute the sub tree, including the join and its operands, with the
empty relation if the qualification of the result of the join is contradictory.
Now let us consider an example of a distributed join. We start from query Q3 that
requires the number SNUM of all suppliers having a supply order. The algebraic
expression of the query over the global schema is
Q3: PJ SNUM (SUPPLY NJN SUPPLIER)
Fig5a shows the canonical form of the query. We recall that the fragmentation of
SUPPLY is derived from the fragmentation of SUPPLIER; i.e., each tuple of
SUPPLY is stored either in fragment SUPPLY1, if it refers to a supplier of San
Francisco, or in fragment SUPPLY2, if it refers to a supplier of Los Angeles.
Applying criterion 1, we push the two unions up beyond the join; thus, we generate
four joins between fragments. We then apply criterion 2, and we discover that two of
them are intrinsically empty because their qualification is contradictory. The empty
joins are those of SUPPLIER1 (in “SF”) with SUPPLY2 (of “LA” suppliers), and
likewise of SUPPLIER2 (in “LA”) with SUPPLY1 (of “SF” suppliers). Thus the
operator tree reduces to that of fig 3.5(b). Assuming that fragments with the same
index are placed at the same site (i.e., that data about SUPPLY are stored together
with data about SUPPLIERS), this operator tree corresponds to an efficient way of
evaluating the query, because each join is local to one site.
PJ
JN DEPTNUM = DEPTNUM
UN UN
[SUPPLY1 SNUM = [SUPPLY2 SNUM = [SUPPLIER1 [SUPPLIER2
SUPPLIER.SNUM AND SUPPLIER SNUM AND CITY =- “SF”] CITY = “LA’]
SUPPLIER.CITY = “SF”] SUPPLIER.CITY = “LA”]
(A) Canonical form of query Q3
UN
PJ SNUM PJ SNUM
JN DEPTNUM = DEPTNUM JN DEPTNUM = DEPTNUM
[SUPPLY1 : SNUM = [SUPPLIER1 [SUPPLY2 : SNUM = [SUPPLIER2
SUPPLIER.SNUM AND CITY = “SF”] SUPPLIER.SNUM AND CITY =
“LA”]
SUPPLIER.CITY = “SF”] SUPPLIER.CITY = “LA”]
(A) Distributed join for query Q4
Fig 3.5: simplification of joins between horizontally fragmented relations[8]
3.3 Using inference for further simplification
Here we want to give examples of how additional information could be used for
simplifying queries. Let us consider again query Q1 that requires the supplier number
of those suppliers having a supply order issued in the North area, for which we have
developed the operator tree in fig 3.4. Assume that the following knowledge is
available to the query optimizer:
1. The north area includes only department 1 to 10
2. Orders from departments 1 to 10 are all addressed to suppliers of San Francisco.
We use the above knowledge to “infer” contradictions that allow elimination of
sub-expressions.
a. From above, we can write the following implications:
AREA = “North” => NOT (10 < DEPTNUM <= 20)
AREA = “North” => NOT (DEPTNUM > 20)
Here we have to apply the selection to fragments DEPT1, DEPT2 and DEPT3 and
evaluate the qualification of the results. By virtue of the above implications, two of
them are contradictory. This allows us to eliminate the sub expressions for fragments
DEPT2 and DEPT3. Thus, the operator tree of fig 3.4(B) reduces to that of fig 3.6a.
b. We then apply criterion 1 for distributing the join; in principle, we would need to
join the sub tree including DEPT1 with both sub trees including SUPPLY1 and
SUPPLY2. But from 1 above, we know that:
AREA = “Nort” => DEPTNUM <= 10
and from 2 above we know that :
DEPTNUM <= 10 => NOT (SNUM = SUPPLIER.SNUM AND SUPLIER.CITY = “LA”)
By applying criterion 2, it is possible to deduce that only the sub tree including
SUPPLY1 needs to be joined. The final operator tree for query Q1 is shown in fig6b.
PJ SNUM
JN DEPTNUM = DEPTNUM
UN PJ DEPTNUM
PJ SNUM, DEPTNUM PJ SNUM, DEPTNUM SL AREA = “NORTH”
[SUPPLY1 SNUM = [SUPPLY2 : SNUM = [DEPT1
SUPPLIER.SNUM AND SUPPLIER.SNUM AND 1 <= DEPTNUM < 10
SUPPLIER.CITY = “SF”] SUPPLIER.CITY = “LA”]
(a)
PJ SNUM
JN DEPTNUM = DEPTNUM
PJ SNUM, DEPTNUM PJ DEPTNUM
[SUPPLY1 SNUM = SL AREA = “NORTH”
SUPPLIER SNUM AND
SUPPLIER.CITY = “SF”
[DEPT1 1 <= DEPTNUM < 10]
(b)
Fig 3.6 : Simplification of an operator tree using inference[8]
3.4 Simplification of vertically fragmented relations
We now turn to the simplification of vertically fragmented relations, which is dual
to the simplification of horizontally fragmented ones. The rationale behind this
simplification is to determine a proper subset of the fragments, which is sufficient for
answering the query, and then to eliminate all other fragments from the query
expression, as well as the joins, which are used in the inverse of the fragmentation
schema for reconstructing the global relations. In particular, if the fragments, which
are used in the inverse of the fragments, which are required, reduce to only one
fragment, there is no need of performing join operations.
We show the simplification with an example. Consider query Q4, which requires
names and salaries of employees. The query on the global schema is simply
Q4: PJ NAME, SAL EMP
The canonical operator tree of the expression is shown in fig 3.7(a). Recall that
EMP is first vertically partitioned into fragment EMP4 and a second fragment, which
is further partitioned horizontally into EMP1, EMP2 and EMP3. We notice that the
attributes of EMP4 include NAME and SAL, which are required by the query. Then it
is possible to answer the query using only the fragment EMP4, and the operator tree
can be simplified by disregarding the other fragments and the join operation. The final
operator tree for query Q4 is shown in fig 3.7(b).
PJ NAME, SAL
JN EMPNUM = EMPNUM
[EMP4 true] UN PJ NAME, SAL
[EMP : [EMP2 : [EMP3 : [EMP4 : true]
DEPPTNUM <= 10] 10 < DEPTNUM <= 20] DEPTNUM > 201]
(a) Canonical form of query Q4 (b) Simplified query
Fig 3.7: Simplification of vertically fragmented relations [8]
Chapter 4
Multi-key Processing 4.1 Introduction
For primary key indexing, each index entry identifies a unique record in the main
file. Any application however, require multikey retrieval, that is the retrieval from a
file of all records having some combination of attribute values. For example, a college
dean might want to genereate a list of all students with
• U.S. citizenship
• Physics or math major
• GPA of at least 3.3
• Student identification number less than 150, 0000
There are likely to be many records in a file with the same value of a particular
secondary attribute. Indexes of various forms are one mechanism for finding them.
Normally secondary index can produce a list of pointers to records having a particular
value of a secondary key.
However there is a price to be paid. Indexes take up space, and if the file is
changed frequently much time may be spent updating secondary indexes. Multi-key
queries involving ranges of attribute values are awkward to deal with using
conventional indexes. They might be better served by the grid file organization.
4.2 Threaded files
In a threaded file a pointer field is associated with each indexed secondary key
field. The value in the pointer field identifies the next record in the file with the same
value of the secondary key. Thus a number of threads run through the file. An entry
points to the first record having the attribute value and acts as a header of a linked list.
Let a file in which records have k attributes and suppose that two of these
attributes are threaded in the manner described. If the first two attributes has N
different values and the second has M different values, then the general form of the
file and indexes would be shown as given bellow
Record
Number
Attribute
1
Next
Pointer
Attribute
2
Next
Pointer
Attribute
3
Attribute
4
Attribute
K
1 - - - - - - -
2 - - - - - - -
3 - - - - - - -
Fig 4.1: general threaded file[21]
A part of a file of car records with one set of threads (for manufacturer) and part
of the corresponding index is shown bellow
Manufacturer index
Ford: 1
↓
VW: 2
↓
BMW: 3
↓
Audi: 11
↓
Honda: 15
Record
No
Manuf. Next
Manuf.
Model Color
Next
Color
License
1 Ford 4 Pinto White 4 HORS4ME
2 VW 6 Bug Red 9 SKIBNY
3 BMW 9 322I Black 6 DADIOUI
4 Ford 5 Mustang White 7 VALEGRL
5 Ford 7 Pinto Blue 8 RATFACE
6 VW 8 Rabbit Black 12 910VCD
7 Ford 10 Pinto White 11 PACMAX
8 VW 12 Rabbit Blue 10 BYE2NOW
9 BMW 16 320 Red 14 CMEGO
10 Ford 14 Mustang Blue 17 DPGURI
11 Audi ? 5000 White 16 OULSK
12 VW 13 Letta Black 18 LKWJE
13 VW ? Bug Green 15 WEIOU
14 Ford 18 Mustang Red 19 SD2332
15 Honda ? Civic Green 20 SDF33
16 BMW 17 320 White ? SDF3
17 BMW ? 322I Blue ? SFSDF23
18 Ford 19 Tempo Black ? TRE344
19 Ford 20 Pinto Red ? 34DRTER
20 Ford ? Mustang Green ? LKJPE
Fig 4.2: threaded file [21]
4.3 Multi-lists
In the multi-list organization the threads in the main file have the same structure
as in the threaded file organization but the index entries are different. Instead of an
index entry pointing simply to the beginning of a thread, it now points to every kth
record on the thread (for some value of k). In effect, we have a number of sub list of
length k and there is a pointer in the index to each subsist. An index entry now has
two links, one to the entry for the next value of the attribute and a second to a list of
pointers to the main file. With this additional information in the index, performance of
merge operations can be speeded up. Here we give an example of multi-list index
using fig 4.2 where the value of k is taken as 3.
Manufacturer index Color index
↓ ↓
Ford :1-7-18-49 White :1-11
↓ ↓
VW :2-12 Red :2-19
↓ ↓
BMW :3-17 Black :3-18
↓ ↓
Audi : 11 Blue : 5-17
↓ ↓
Honda : 15 Green : 13-25-40-28
Fig 4.3: Multi-lists [21]
4.4 Inverted files
Here the initial threaded file represents one extreme of the multi-list organization
with k = ∞. The other extreme is when k = 1; in this case the index points to every
record with a particular attribute value. This type of multi-list organization is known
as inverted file. Virtually all the commercially available systems are based on inverted
file designs.
4.4.1 STAIRS: An Application of Inverted Files
IBM’s STAIRS (Storage And Information Retrieval System) is a powerful
document retrieval system. Users can retrieve documents based on their content. For
example, they can retrieve documents containing an arbitrary word, or those
satisfying a complex Boolean expression of words.
The STAIRS system has this capability because it indexes every word occurrence
in the text, in contrast to most document systems, which index a few selected
keywords. STAIRS can thus be classified as a full text document retrieval system.
Here we give a brief description of the file structures that make retrievals efficient and
show how queries are answered. The file structures we describe are simplified
versions at the actual STAIRS structures.
Matrix
↓
Dictionary
↓
Occurrence file
↓
Index
↓
Documents
Fig 4.4: STAIRS file hierarchy [21]
4.4.2 File structure:
The STAIRS system contains five levels of data structures files as depicted in
fig4. The lowest level in the structure, the documents file, contains the machine-
readable documents. The only change from the conventional representation of a
document is that each paragraph is tagged with a label such as TITLE, TEXT,
ABSTRACT, and so on. In addition, the document contains end-of-sentence codes
that the system can recognize.
The next level in the structure, the index, has one entry for each document. The
entry contains information such as a pointer to the document, protection codes, and
date of entry into the system. The three file levels above the index refer to a document
by a unique document number: the number of its entry in the index.
The occurrence file contains one record for each word occurrence in the document
collection. The information recorded for each word occurrence is:
• Document number
• Paragraph code
• Sentence number
• Position within sentence
The entries in the occurrence file are ordered so that all records for a particular
word are contiguous. Within this grouping, records are stored in the order of the four
fields listed above.
The dictionary contains an entry for each different word in the document
collection, including such common words as ‘the’, ‘of’, and ‘in’. Summary
information, such as the numbers of times the word occurs and the number of
different documents in which it occurs, is stored together with the word.
In large dictionaries in book form usually have a thumb index, which enables the
user to find an alphabetical section rapidly? The matrix takes this one step further. It
has 26*27 entries, each of which identifies the start of a section of the dictionary for
words beginning with a particular pair of letters. In fact matrix eliminates the need to
store the first two letters of words in the dictionary. This key compression saves a
certain amount of space.
Fig5 represents a small part of the top three levels of an example file collection.
We assume in this example that in the document collection, “macabre” is a
alphabetically the first word starting with the letters “ma”. Also, we assume that the
word “mainframe” occurs a total of 109 different times and in 20 different documents.
Chapter 5
Index implementation 5.1 Introduction
With the multi-list and inverted file indexes there is a problem of maintaining
variable-length lists for each attribute value. This is one of the major problems of
multi-list indexing. Here we consider two alternatives to simple lists: bit vectors and a
general graph structure.
5.2 Bit vectors
A bit vector in the context of indexes is an array of two-valued objects having as
many elements as there are records in the main file. Each element indicates whether
or not the corresponding main file record has a particular attribute of the car file of
fig2.
Figure 5.1: Bit Vector Index [22]
5.3 Graph structure
We can save a certain amount of space in a structure by combining those list
elements that point to the same main file record. Thus each main file record is
represented in an inverted directory file by a single node that is on as many lists as
there are indexed attributes. A node will have, for each indexed attribute, a pointer to
the next node representing a main file record with the same value of that attribute. It is
Manufacturer
Ford
VW
BMW
Honda
101110000110101111000011100
110010101001110111000000110
101101010101010001100000001
101010101011111100000110101
convenient if each node also contains a pointer back to the owner of the list, that is,
the index entry for the particular attribute value. Fig6 shows what a node might look
like. Fig7 shows a small part of the graph structure for our car file. Only nodes for
records 1, 4 and 5 are shown.
Fig 5.2: General graph node[22]
Pointer to main file Pointer to owner node attribute 1 Pointer to next node with same value of attribute 1
Fig 5.3 Partial Graph Structure [22]
To find green fords we choose, arbitrarily, one of the two attributes, for example
manufacturer. We follow the manufacturer pointers from the first node pointed to by
“Ford”. At each node visited we check the owner of the color attribute to see if it is
green.
Since the nodes are distributed all over the file, we may have to access the entire
file to traverse the path. This takes a lot of time to find a specific query. Grid File
Organization can eliminate this problem.
Manufacturer
Ford
VW
BMW
Model
Pinto
Bug
Mustang
Color
Red
White
Black
To main file
To main file
To main file
To node for record to next mustang
To node for record to next white car
To node for record to next Ford
To node for record to next Pinto
To node for record to next Black
5.3.1 Comparison of Bit Vectors and Graphs
Bit vectors may at first appear to have higher storage costs than graph structures.
However, suppose that the main file contains M records, that P different attributes are
indexed, and that the attributes have an average of N different values. The storage
required for the bit vectors is
N*M*P bits
Assuming that we are combining nodes as described above, the storage for the
comparable part of the graph structure (M nodes) is
M*(2*P+1) pointers, which equals (2*M*P)+M pointers
Roughly speaking, if the number of different values of an attribute is less than the
number of bits required to hold two pointers, then a set of bit vectors occupies less
space than the comparable graph structure. A hybrid system might be a preferred
compromise; bit vectors would be used for attributes with few different values and
lists would be used for attributes with many different values.
The biggest advantage of bit vectors is the speed with which simple set operations
can be performed on conventional hardware. Most computers have machine-level
instructions for performing logical operations on bit patterns. In contrast, list merging
is comparatively slow.
5.3.2 Index maintenance
Secondary indexes, like primary indexes, must reflect the contents of the main file
at all times. Although there is only one primary index, there may be several secondary
indexes. Thus maintenance of correct index entries may become a big overhead. Here
we consider how bit lists and graph structures compare in the amount of maintenance
required.
5.3.2.1 Updating
An inverted file must be updated in three cases: when we insert a new record into
the main file, when we delete a record, or when we change the value of a secondary
key of an existing record. In the case of an insertion or a deletion, all indexes for the
file have to be modified. In the case of a single field modification, one index at most
has to be updated.
To delete a record from the main file we could simply mark it deleted. The
alternative is to rewrite the file and omit the deleted record. The marking operation is
a logical rather than a physical deletion. Logical deletion is simpler. A disadvantage
of physical deletion is that records may change position in the file and thus require
changes in pointers to them. An advantage of physical deletion is the reduction in list
length; this makes subsequent traversals shorter. For some insertions we may be able
to reuse logically deleted records in the main file, in which case there is no problem in
updating a bit vector.
If we change the value of an attribute the changes in the bit list are vary small. We
simply clear a bit in one list and set it in another. When we modify an attribute in a
graph structure, on the other hand, we must release the node from one inverted
attribute list and assign it to another.
Both the insertion of a new record and a change in an attribute value might
introduce an attribute value not previously represented in the file. In the case of a bit
vector, we must create a new vector with exactly one bit set. In the case of the graph
structure, there wil be a new list with exactly one node.
5.3.3 Reliability
Secondary storage tends to be more vulnerable than primary storage to data
corruption. Some of the data structures we have described above can be rendered
useless if a critical pointer is damaged. We thus therefore consider methods of making
the structures more robust, that is, less likely to be damaged irreparably. A general
technique might be to provide two or more paths to any record. A file maintenance
program can check the integrity of access paths. If a damaged path is detected; we can
possibly use another route and repair it. This recovery principle suggests that a
double-linked circular list is preferable to a single-linked list. 5.4 Grid Files
Nievergelt, Hinterberger, and Sevcik describe a secondary key accessing
teachnique-using grids that performs well on both stable and volatile files. We
describe the briefly here.
5.4.1 Design Aims
The design aims of the grid file organization are fourfold:
1. Point queries. The processing of a completely specified query should require no
more then two disc accesses. A completely specified query (or point query) is one
in which a single value is specified for each key attribute. An example from the
car file is: find Manufacturer = Ford, Color = Black, License = 1GWN821, Model
= Bug
2. Range queries: Processing of range queries and partially specified queries should
be efficient. Two examples of such queries are: find Manufacturer = VW, Model
= Bug, D < license < X: find Manufacturer = Ford, Color = Green
3. Dynamic adaptation: The file structure should adapt smoothly to insertions and
deletions.
4. Symmetry: All key fields, whether primary or secondary, should be treated
equally.
5.4.2 Ideal Solution
Assume that records have k keys. Consider the k-dimensional space defined by
the k sets of attribute values. Modifying the bit vector idea given above, we can
conceive of a k-dimensional bit matrix I have different values. If a particular element
of the k-dimensional matrix is set to 1, this indicates that a record exists with the
corresponding set of k attribute values. If the bit is 0, then no such record exists. Note
that the matrix of bits is therefore a complete representation of the set of records.
This organization satisfies the four design aims above, although we have assumed
nothing about how it might be stored. Processing point queries involves examination
of a single element. Processing range queries involves processing all elements in
particular j-dimensional matrix j <= k. Insertions and deletions are carried out by
setting and clearing appropriate single elements of the matrix. All key fields are
treated equally.
The k-dimensional matrix, however, is an ideal rather than a practical file
organization. In practice the matrix would be far too large to store. If a file has records
with 4 keys and each attribute has 1—different values, the matrix will have
100,000,000 elements. The grid file organization that we discuss next is in some ways
an approximation of the matrix ideal.
5.4.3 Practical Grid File Implementation
In the grid partitioning sets of attribute values reduces file organization the size of
the matrix. For example, we could partition the set of colors into four subsets. If we
regard attribute values as character strings, we might have
Color < F
F <= Color < K
K <= Color < Q
Q <= Color
Thus the color lemon, for example, falls into the third partition (K <= lemon < Q).
The partition points are held in linear scales. The set of k linear scales, one for each
attribute, defines a grid on the k-dimensional attribute space. The space is thus
divided into grid blocks. The number of grid blocks is much smaller than the number
of elements in the matrix. What we have lost, however, is the one-to-one
correspondence between elements of the grid/matrix and possible records.
In the grid file organization, records are stored in buckets. Buckets have a fixed
size, but there can be arbitrarily many of them in a file. The dynamic assignment of
buckets to grid blocks is maintained in the grid directory. The grid directory consists
of linear scales and a grid array, where each element contains a pointer to a bucket,
grid array elements, which form a k-dimensional rectangle, any point to the same
bucket in the file where records with the corresponding attribute values are stored.
We can represent the partitioning of the Color attribute described above by the
following linear scale
Color (F, K, Q)
Suppose that the other three attributes are partitioned similarly and that the linear
scales are
Manufacturer (G, R)
Model (C, H, N, T)
License (B, M, S)
Consider the record with
Manufacturer = Ford, Model = Pinto, Color = Blue, License = BBC1500
The partitions into which the attribute values fall are 1, 4, 1 and 2 respectively.
Therefore, the bucket pointed to by
Grid-array [1, 4, 1, 2]
Is the only place where the record would be stored?
The grid array of pointers is normally so large that it must be held in secondary
memory. On the other hand, the linear scales would normally fir into main memory.
Next we consider how well the grid file organization achieves the four design aims of
point queries range queries, dynamic adaptation, and symmetry.
5.4.4 Performance of Grid Files
Point queries: In response to a point query, each of the k specified attribute values
is first transformed into a grid index using the appropriate linear scale. The element of
the grid array selected by the set of k indexes can now be fetched from disc.
Nievergelt, Hinterberger, and Sevcik make a number of suggestions for implementing
the grid array. The calculations involved in mapping a rectangular k-dimensional
array onto linear memory are not complex. The address of a particular element is easy
to compute, and the element can be fetched in one access. A second disc access
fetches the bucket pointed to from the array element. Thus the first design aim is
achieved.
5.4.4.1 Range queries
The second design aim is to answer range queries efficiently. To satisfy this aim it
must be possible to move efficiently along an arbitrary axis of the grid array. That is,
given the address of a particular element, it must be easy to compute the address of
the next or previous element in any of the k-dimensions. For example, to satisfy the
range query
Manufacturer = VW, Model = Bug, D < license < X
We need to process records in buckets pointed to by elements in the rectangle
Grid array [3, 1, 1…4, 2…4]
A linked list implementation of the matrix would satisfy this requirement. An
array implementation enabling direct access of an element given its indexes would
also be suitable.
5.4.4.2 Dynamic adaptation
The third design aim is that the organization should adapt smoothly to insertions
and deletions. Let us consider these in turn.
5.4.4.3 Insertions
If a record must be inserted into a bucket that is already full, then a new bucket is
allocated to the file and records are distributed between the two buckets. There are
two cases to consider: when only only pointer points to the full bucket.
If the full bucket is pointed to form more than one element of the grid array, we do
not need to make changes to the partitioning. Records are distributed between the two
buckets according to current partitioning, and some of the pointers change from the
old to new bucket.
Suppose, for example, that the grid element representing records with
Color < F
G <= Manufacturer < R
H <= Model < N
License < B
And the grid element representing records with
Color < F
G <= Manufacturer < R
N <= Model < T
License < B
Both point to the same bucket and that this bucket overflows. A new bucket is
allocated to the file, and one of the two elements of the grid array is changed to point
to it. Records in the overflowing bucket are distributed between the two buckets
according to whether the value of the model attribute is less than N.
If only one grid element points to a bucket, the grid must be refined. One of the
sub ranges represented by the bucket contents must be divided. A new partition point
is added to the appropriate linear scale. One bucket is assigned to each half of the
original grid element, and the records are distributed according to the new
partitioning.
Suppose that after further insertions there is overflow in the bucket pointed to by
the element representing records with
Color < F
G <= Manufacturer < R
N <= Model < T
License < B
Assume further those only one-element points to this bucket. We therefore need to
split one of the sub ranges. Choosing arbitrarily, the Manufacturer dimension, we
could insert a partition at M. the corresponding linear scale is now
Manufacturer (G, M, R)
The number of elements in the grid array increases by 33% because the
manufacturer dimension now has four rather than three sub ranges. Most of the new
elements will point to an existing bucket. For example, the element representing
records with
J <= Color < K
G<=Manufacturer<M
Model < C
M <= License < S
Will point to the same bucket as the element representing records with
F <= color < K
M <= Manufacturer < R
Model < C
M <= License < S
We need to allocate a new bucket to the file and distrib ute records from the
overflowing on. The two buckets will now be pointed to by the array elements
representing
Color < F
G <= Manufacturer < M
N <= Model < T
License < B
And
Color < F
M <= Manufacturer < R
N <= Model < T
License < B
5.4.4.4 Deletions
To maintain reasonable storage utilization, two candidate buckets might be
merged if their combined number of records falls below some threshold. The records
would be moved into one of the buckets and pointers to the other reassigned to it. The
empty bucket would be de-allocated from the file. Note that not every pair of buckets
can be merged. Only elements that form a k-dimensional rectangle can point to a
particular bucket.
5.4.4.5 Symmetry
The symmetry of the matrix ideal is preserved in the grid organization. There is no
performance difference between primary and secondary indexing because all indexed
attributes are treated in the same way.
Chapter 6
PROPOSED DISTRIBUTED
DATABASE INDEX 6.1 Introduction
In previous chapters we have discussed index techniques and their uses in the
databases. Indices are associated with the main data file. They facilitate the Database
Management Systems to access records in the data file faster and in a random fashion.
There are different types of index structures and algorithms. Each has some
advantages and disadvantages over the others. One structure may be suitable in one
context but may not be suitable in another context. For example B, B* and B+ trees
are often used for implementing dynamic index in primary key implementation. But
for secondary keys such as multi-key indexing the methods are Bit Vectors, Graph
Structures, and Grid File Organizations etc. We have discussed about those methods
and their advantages and disadvantages in previous chapters. Now we will show how
we can improve searching in distributed database management systems.
6.2 Distributed Database
In recent years, distributed databases have become an important area of
information processing, and it is easy to foresee that their importance will rapidly
grow. There are both organizational and technological reasons for this trend.
Distributed databases eliminate many of the shortcomings of centralized databases
and fit more naturally in the decentralized structures of many organizations.
A distributed database is a collection of data, which belong logically to the same
system but are spread over the sites of a computer network.
For example, consider a bank that has three branches at different locations. At
each branch, a computer controls the teller terminals of the branch and the account
database of the branch. Each computer with its local account database at one branch
constitutes one site of the distributed database; a communication network connects
computers. During normal operations the applications, which are requested from the
terminals of a branch, need only to access the database of that branch. These
applications are completely called local applications. An example of a local
application is a debit or a credit application performed on an account stored at the
same branch at which the application is requested. Some applications are called global
applications or distributed applications. A typical global application is a transfer of
funds from an account of one branch to an account of another branch. This application
requires updating the databases at two different branches.
Therefore, a distributed database is a collection of data distributed over different
computers of a computer network. Each site of a network has autonomous processing
capability and can perform local applications. Each site also participates in the
execution of at least one global application, which requires accessing data at several
sites using a communication subsystem. Figure given below shows a typical
Distributed Database.
6.3 Finding records in a Distributed Database
At present distributed databases are inefficient in locating records since it is not
using any global index structure. For example, if we have a book data file in a
distributed database, the single book data file should be fragmented into several data
files and these fragments should be allocated in different sites of the distributed
database. The fragments information will be stored in the fragmentation schema, and
the information regarding the allocation of fragments to the sites will be stored in the
allocation schema. When a query is searching a book by a particular author it will be
fragmented into sub queries according to fragmentation and allocation schemas
because the fragmentation schema will tell the number and the names of the
fragments of the data file and the allocation schema will tell about the sites where to
get the said fragments. Using this information Distributed Database Management
Systems submit the fragmented queries into sites and collect the results from different
sites. Some sites that don’t contain the query information will result empty set. So it is
a waste of time to submit the fragmented queries in those sites.
If we know before hand which sites contain the required information and submit
fragmented queries only in those sites then it will faster and more efficient than
before. We can achieve this goal by implementing global index.
6.4 PROPOSED DISTRIBUTED INDEX
6.4.1 INTRODUCTION
The reason for providing indexes is to obtain fast and efficient access to data.
Indexing is a data structure based technique for accessing records in a file. Multi-key
indexing is often graph structured or grid file organized. Though local database has
index file to search a record efficiently distributed database has no such opportunity.
Our aim is to organize the whole thing so that global queries can be executed with
more efficiently and more fast.
6.4.2 Local Index Architecture
In local Database Management Systems primary key index is implemented in
B/B*/B+ trees. Grid File Organization often implements by Bit Vector, Graph
Structure or multikey indices. Normally Graph Structure and Grid File are used
widely.
6.4.3. Comparison among Bit Vector, Graph Structure and Grid File
Organization
6.4.3.1 Advantages and disadvantages of Bit vector
If we implement index on one attribute then we need a two-dimensional index
vector as shown in chapter 5. This bit vector is efficient because of the speed with
which simple set of operations can be performed on conventional hardware. But if we
want to implement a multi-list index with bit vector then a three dimensional bit
vector will be necessary, where the third dimension will represent the fields on which
the index is to be implemented. For example let we want to make a multi-list index on
the attributes Manufacturer, Model and Color of a car. Then the third dimension or the
z-axis of the bit vector will represent the bit vectors of Manufacturer, Model and
Color. Now if we want to know the query where Manufacturer = Ford, Model =
Mustang, and Color = Green, we have to access the corresponding bits from the bit
vector and to find all three bits are to be 1. But it is not easy to access those three bits
at the same time. The same problem we face when we delete of update a record.
Again different attribute has different no of attribute values. So the bit vector is not
always equilateral. It also makes maintaining the bit vectors difficult. These problems
are solved by graph structures.
6.4.3.2 Advantages and disadvantages of Graph Structure
To eliminate the problems of bit vector, in graph structure the record information
is represented in concise way. As described in previous chapters, if we want to find a
query we have to traverse the nodes through a single path of any one attribute. As the
attributes may be distributed all over the index file this requires to check all the nodes
throughout the entire index file. This may cause to access the index file several times,
which make the process inefficient. These problems are solved in grid file
organization where a particular record can be found only in two disk accesses.
6.4.4 Distributed Index Architecture
6.4.4.1 Introduction
To provide efficient access to the data we proposed a distributed index concept.
Distributed index is also a data structure based index comprising of two types of index
structures. One is Global Index (GI) and the other is Local Index (LI). Figure 6.2
shows the proposed distributed structure.
Fig 6.2: Architecture of Distributed Index [8]
GI is created and maintained by distributed database component (DDB) of
distributed database management systems (DDBMS). LI is created and maintained by
local database management component (DB) of DDBMS. Our study shows Bit
Vector, Graph Structure and Grid File Organization all have advantages in some way
and have disadvantages in other way. For this reason we preferred a way which uses
the techniques of Bit Vector, Graph Structure and Grid File Organization for GI. LI
has been imple mented by grid file. For every site there is a Local Index (LI), which
has been created, updated and used independently. Like other local database
management components LI enjoys autonomy in each site. There must be a single
global index (GI) for a distributed index. GI is created, updated and used based on
local indexes. All the local indexes are perfectly mapped with the global indexes.
When a record is searched in a distributed database, GI is used first to determine
which LI needs to be used to find the data. After selecting the right LI it is used to
access records in the corresponding site. In this way, distributed index ensures
efficient access to the data in a distributed database. If a record is inserted, deleted or
updated in a site and if it causes a new combination of fields or deletes a record
completely in any other sites then the record information is passed all other sites. The
other sites update their own global index according to the record information.
Proposed Global Index (GI) is a combination of Bit Vector, Graph Structure and
Grid File. In a grid file the actual records of database is stored in buckets as described
in previous chapters, and the index file is created to access those buckets. Introducing
Grid Dictionary accelerates this access. The Grid Dictionary consists of linear scales
and a grid array, where each element contains a pointer to a bucket. On the other hand
in Graph Structure all the records are stored in index file concisely as graph nodes,
where only two pointers for each indexed attribute, the forward pointer and the back
pointer, and an original pointer to the record of main file is stored.
Our main goal is to optimize query submission to database sites so that
unnecessary submission is not made. Hence we want to know every possible record of
other sites. This is possible in the form of graph nodes. For example if there are M no
of records in all sites but N (N < M) no of different records, then we have to keep
information of N records. Let us clarify it fully by an example. We have a database of
Manufacturer, Model, Color and License of cars at two different sites where multi key
index is created on Manufacturer, Model and Color fields. The site records are given
below.
Records in site1:
Rec. no Manuf. Model Color License
1 Ford Pinto Green 23023234
2 VW Civic White 23424244
3 BMW Bug Red 43543535
4 Ford Mustang Black 65435435
5 BMW Mustang White 45645654
6 Honda Tempo Green 23432543
7 VW Civic White 54654645
8 BMW Bug Red 34543543
9 Ford Pinto Green 54654664
10 Honda Tempo Green 54654654
Here, there are 10 records but 6 different records according to three indexed fields
as record 1 & 9, 2 & 7, 3 & 8, and 6 & 10 have same value in three fields.
Records at site2:
Record no Manuf. Model Color License
1 Ford Mustang White 23432424
2 VW Civic White 34655466
3 Ford Pinto Green 45654646
4 BMW Pinto Green 65765353
5 Ford Mustang White 32984983
6 Ford Pinto Green 56765756
7 Honda Tempo Red 54366547
8 VW Civic White 54765466
9 BMW Pinto Green 45645664
10 Honda Tempo Red 45654654
Similarly here, there are only 5 different records. Again in both sites there are only
9 no of different records and we have to keep information about these 9 records in our
global index (GI) file.
Now our aim is that, if we want to find a record with Manuf = Honda, Model =
Tempo and Color = Green then we will submit the query only in site 1 as site 2 has no
such records, but if we want to query the record with Manuf = Ford, Model = Pinto
and Color = Green, then we will submit the query to both sites.
To achieve this goal we will store the 9 different records in global index file as a
form of graph nodes and the graph nodes will be stored as a fashion of Grid File
Organization. Here the graph nodes will consists of three back pointers and a bit
vector, where the bit vector gives the site address of that specific record and it
replaces the original pointer of records in data files. Here the graph nodes will be like
the following figure:
Bit vector
Back Ptr1
Back Ptr2
Back Ptr3
The corresponding graph nodes of the two records with Manufacturer = Honda,
Model = Tempo, Color = Green and Manufacturer = Ford, Model = Pinto, Color =
Green are given below.
This record exists in site 1 this record exist in both sites
In global index file these records will be stored as the original records of database
are stored in grid file. Thus by partitioning the entire range of different values of
individual fields into linear scales we will access the bucket of these records by grid
array and then find the desired records searching through the nodes linearly. The
whole process is done like Grid File Organization as if the nodes are now treated as
original records. Let us clear the whole thing using the example of the above two
database sites.
The different values of the three attributes are given in the following table
Manufacturer Model Color
BMW Bug Black
Ford Civic Green
Honda Mustang Red
VW Pinto White
Tempo
Now we could partition the three set of attributes into subsets. If we regard
attribute values as character strings, we might have
Manufacturer < G , G <= Manufacturer
Model < K, K <= Model < R, R <= Model
Color < H, H <= Color
Thus the color Black falls into the first partition (Black < G). Similarly Model
Pinto falls into the second partition (K <= Pinto < R) and Manufacturer Ford falls into
the first partition (Ford < G). The partition points are held in linear scales. The set of k
10
11 Site bit vector
Honda
Tempo Green
Site bit vector
Ford
Pinto Green
linear scales, one for each attribute, defines a grid on the k-dimensional attribute
space. The space is thus divided into grid blocks.
Now, consider the first record of site1 database with
Manufacturer = Ford, Model = Mustang and Color = Black
The partition into which the attribute values fall are 1, 2 and 1 respectively.
Therefore the bucket pointed to by Grid-array[1,2,1] is the only place where the
record would be stored.
The grid array of pointers is normally so large that it must be held in secondary
memory. On the other hand, the linear scales would normally fit into main memory.
The grid array contains the pointers of the buckets where the records are stored. The
grid array looks like figure 6.3.
Fig 6.3: Grid-Array pointing to buckets [15]
[1, 1, 1]
[1, 1, 2]
[1, 2, 1]
[1, 2, 2]
[1, 3, 1]
[1, 3, 2] . . . . . .
[2, 3, 2]
Bucket 1
Bucket 2
Bucket 3
0 1 2 3 4 5
Grid Array
Bucket 12
* *
So the record with Manufacturer = Ford, Model = Mustang and Color = Black
falls into bucket3 among with other records that belongs to the group of grid-array [1,
2, 1]. We can find the group [1, 2, 1] in the grid array by using the following formula:
In general if the grid array is [i, j, k] and there are P, Q and R no of subsets in
linear scales then the grid array index will be (i-1)*Q*R + (j-1)*R + (k-1). Here in our
example the value of i = 1, j = 2, k = 1, P = 2, Q = 3 and R = 2. So the index value of
grid array is (1-1)*3*2 + (2-1)*2 + (1-1) = 2.
After finding the bucket pointer from the grid-array we access the bucket from the
file and search linearly through the entire bucket. The record can exists only in that
bucket. While finding, we visit each node in the bucket and check the back pointer for
real attribute values. Here the organization of Bucket 3 is given in figure 6.4.
Fig 6.4: Proposed Global Index File Organization
From the above figure we see that we will find the record with Manufacturer =
Ford, Model = Mustang and Color = Black at the second position of the bucket. At the
node the bit vector is ‘10’ that is the record exists only in site1 but not in site2. Hence
we will submit the query for the record in site1 but not in site2.
Manufacturer
Ford
VW
BMW
Model
Pinto
Bug
Mustang
Tempo
Civic
Color
White
Black
Green
Red
11
10
01
00
00
00
Bucket 3
6.4.4.2 Distributed Global Index Searching (GI Searching)
Searching point query
When searching for a record with a given key value we start searching in global
index first to find out the right local index. We start searching the global index by
finding the grid array index using the linear scales upon which the field values are
divided. When we find the grid array index we find the bucket pointer from the grid
array and access the corresponding bucket. Then we search for the record throughout
the entire bucket with the help of back pointer of the nodes. If the record is found then
we get the site bit vector from the node and send the query to the sites in which the
record actually exists.
Search Point Query in GI ( searchKey )
{
get the grid array index using linear scales
find the bucket pointer from grid array using the index
access the bucket from index file
search the desired record in the bucket
if the record is found get the bit vector of that record
Submit the query according the bit vector where the corresponding record
actually exists
}
When the subqueries are submitted into local sites, the local sites search the
records using their own local index. Here the actual records are stored in grid file. So
its local index consists of grid dictionary. Using this LI, local database management
system searches the record and returns the result to the site that issue the original
query. Here the search technique follow the same as GI search for a particular graph
node in global index file. The LI search technique is given below:
Search Point Query in LI ( searchKey )
{
get the grid array index using linear scales
find the bucket pointer from grid array using the index
access the bucket from main file
search the desired record in the bucket
the record is found and return that to original site that issued the query
}
Searching range query
Two examples of such queries are: find Manufacturer = VW, Model = Bug,
D < license < X: find Manufacturer = Ford, Color = Green
Search Range Query in GI ( searchKey )
{
get the grid array index range using linear scales
find the bucket pointers from grid array using the index
access the buckets from index file
search the desired record in the buckets
if the record is found get the bit vector of the records
do OR operation among the bit vectors
If all bit become one
Submit the query all the sites
Else
Submit the query which sites have those records
}
Algorithm for searching in Global Index of a Distributed Database
6.4.4.3 Global Index Insertion (GI Insertion)
Let a record with Manufacturer = Ford, Model = Mustang, and Color = Black is
inserted into local database at site 2. Each time a record is inserted it is checked
whether it is a new combination of values of indexed attributes. Here this record is
new for site2. So it will send the values of this record to other sites like site1. Site1
will now locate the bucket where this record information could be stored. As
mentioned above this record falls into Grid Array [1, 2, 1] and from this we know that
the record will be stored in bucket 3. Now in bucket3 the record is searched. Here we
find the record in position two in the bucket. Now what we do is to turn on the bit
value of that node for site2 and save the node in the bucket again. The site bit vector
will now look like 11.
Let us now try another example. We want to insert a record with Manufacturer =
Audi, Model = Pinto, and Color = White. After inserting the record into site1 it will
be found as a new combination of indexed attribute values. So its information will be
passed to other sites to maintain GI. As calculated above this record information will
be stored in bucket3. Bucket3 is searched whether this record already exists or not. In
that case this record doesn’t exists. So a new node is created with attribute values
pointers pointing to their original value, and its site bit vector is made 10 as the record
exists only in site1.
While creating a new node in any bucket it could be found that the bucket is
already full. In that case the bucket should be split into two buckets and records will
be distributed between those two buckets. The splitting process depend on two cases.
1. when only one pointer points to a bucket
2. when more than one pointer points to a bucket
let us now clarify the process with two examples. Before this we assume that, at
present the elements of Grid Array ([1, 1, 1] to [2, 3, 2]) points to only one bucket.
Case 1
Let, while inserting a record with Manufacturer = BMW, Model = Pinto, and
Color = Black we find the corresponding bucket is full. According to linear scales this
record falls into the bucket pointed by the Grid Array [1, 2, 1] i.e. the 3rd bucket and
only one pointer points this. In that case one of the sub ranges represented by the
bucket contents must be divided. Choosing, arbitrarily, the Manufacturer dimension,
we could insert a partition at C. The corresponding linear scale is now
Manufacturer (C, G), i.e.
Manufacturer < C, C <= Manufacturer < G, G <= Manufacturer
Previously the number of Grid Array element was 2*3*2 or, 12, but now the
number of Grid Array element is 3*3*2 or, 18. The number of elements increases by
33%. Most of the new elements will point to an existing bucket. Here the new Grid
Array structure is given below:
Fig 6.5: Insertion of record in (case 1)
Previously the bucket1 was pointed only by one pointer Grid Array [1, 1, 1] but
now bucket1 is pointed by two pointers, Grid Array [1, 1, 1] and Grid Array [2, 1, 1].
Previously the records, which fall into group
Manufacturer < G
Model < K
Color < H
Now fall into two groups. The groups are
[1, 1, 1]
[1, 1, 2]
[1, 2, 1]
[1, 2, 2]
[1, 3, 1]
[1, 3, 2]
[2, 1, 1]
[2, 1, 2]
[2, 2, 1]
[2, 2, 2]
[2, 3, 1]
[2, 3, 2]
[3, 1, 1] . . . . .
[3, 3, 2]
Bucket 1
Bucket 2
Bucket 3
0 1 2 3 4 5 6 7 8 9 10 11 12 . . . . . . .
17
Grid Array
Bucket 12
Bucket 13
* * * * *
Manufacturer < C, Model < K, Color < H
And
C <= Manufacturer < G, Model < K, Color < H
But, since the records of both groups were in bucket1, now both the pointers point
to bucket1. Similarly both Grid Array [1, 1, 2] and Grid Array [2, 1, 2] point to
bucket2 and so on.
We actually split bucket3, which was pointed by Grid Array [1, 2, 1] previously.
Now we allocate a new bucket, bucket13, as there were 12 buckets previously, and
distribute the records of bucket3 to bucket3 and bucket13 according to the following
two groups:
1. Manufacturer < C, K <= Model < R, Color < H
2. C <= Manufacturer < G, K <= Model < R, Color < H
Let we keep group1 in bucket3 and group2 in bucke13. So Grid Array [1, 2, 1] still
points to bucket3 and Grid Array [2, 1, 1] now points to bucket13.
Case 2
Let us now insert a record that falls into bucket2, which is pointed by two
pointers, Grid Array [1, 1, 2] and Grid Array [2, 1, 2]. Now if we find that the bucket
is full then we don’t split the linear scales. Rather we will allocate a new bucket,
bucket14, and distribute the records of bucket2 into those two buckets according to
the following groups:
1. Manufacturer < C, Model < K, H <= Color
2. C <= Manufacturer < G, Model < K, H <= Color
If we keep group1 in bucket2 and group2 in bucket14 then Grid Array [1, 1, 2]
still points to bucket2 and Grid Array [2, 1, 2] now points to newly created bucket,
bucket14. The related picture is drawn below for further clarification:
Fig 6.6: Insertion of record in (case 2)
[1, 1, 1]
[1, 1, 2]
[1, 2, 1]
[1, 2, 2]
[1, 3, 1]
[1, 3, 2]
[2, 1, 1]
[2, 1, 2]
[2, 2, 1]
[2, 2, 2]
[2, 3, 1]
[2, 3, 2]
[3, 1, 1] . . . . .
[3, 3, 2]
Bucket 1
Bucket 2
Bucket 3
0 1 2 3 4 5 6 7 8 9 10 11 12 . . . . . . .
17
Grid Array
Bucket 12
Bucket 13
* * * * *
Bucket 14
Insert LI ()
{
Insert the record in local database
If the record is new
send it to all other sites
Insert GI ( inRec )
}
Insert GI ( inRec )
{
Search GI ( inRec )
If inRec exists in global index
then simply turn on the bit of the corresponding site
Else
insert the inRec into the bucket
if bucket is full
find whether the bucket is pointed by one pointer or more than
one pointer
if the bucket is pointed by only one pointer
randomly select any one of the linear scales
divide it into its middle
create a bucket and distribute the records according to
new linear scales
create a node in the corresponding bucket
turn on the bit of the corresponding site
save the node in the bucket
else if the bucket is pointed by more than one pointer
create a new bucket
make necessary change in grid array pointers
distribute records according to grid array pointers and
their subdivisions
create a graph node in the appropriate bucket
turn on the bit of the corresponding site and save it
}
Algorithm for inserting records in Global Index of a Distributed Database
6.4.4.4. Global Index Deletion (GI Deletion)
When a record is deleted from a local database it is checked that whether there is
any more such records in that database. If there are no such records then the
information is passed to other sites to delete the record information from their GI.
When a site receives record information to delete from its global index (GI), it
searches the corresponding node in buckets and if it finds the node it turns off the
corresponding site bit from the site bit vector of that node. Now it checks whether all
the bits of the site bit vector is zero. If so, then there are no such records in any other
sites. So it deletes the node from the bucket.
To maintain reasonable storage utilization, two candidate buckets might be
merged if their combined number of records falls below some threshold. The records
would be moved into one of the buckets and pointers to the other reassigned to it. The
empty bucket would be de-allocated from the file. Note that not every pair of buckets
can be merged. Only elements that form a k-dimensional rectangle can point to a
particular bucket.
Delete LI ()
{
delete a record from local database
if there is no such record anymore
to delete the record information from all other sites call the function
DeleteGI ( outRec )
}
Delete GI ( outRec )
{
Search GI ( outRec )
If the outRec is found in the bucket
Turn off the bit of the corresponding site
If all site bits is zero
delete the record from bucket
If the no of records in bucket falls below some threshold
Move records into one bucket
Re-assign the other pointers to it
}
Algorithm for deleting records in Global Index of a Distributed Database
6.5 Performance evaluation
In Global Index structure the techniques of Bit Vector, Graph Structure and Grid
File Organization are used. If there are P no of different values of one attribute, Q no
of different values of another attribute and R no of different values of the other then
we have to keep information at most P*Q*R no of records. This is less than the no of
total actual records in all sites. Previously in Graph Structure two pointers are used for
each attribute, but now only one pointer is used. Again the bit vector of site addresses
replaces the space of original record data file pointer. So the total amount of space
required for a graph node is almost half than the previous one. Since the graph nodes
or so-called records are stored in GI file in a fashion of Grid File Organization we
achieve the goal of searching a record in two disc file access for point queries. It also
facilitates searching range queries and dynamic adaptation such as insertion, deletion
and updating. Since only one bit is used to keep information whether a record exists in
a site or not it reduces the amount of space and also speeds up the execution.
The performance of global index is dependent on how the records of original
database are distributed over the sites. For example, a record, with Manufacturer =
Ford, Model = Pinto and Color = Black, if exists in 2 or 3 sites of total 8 sites then we
gain a considerable efficiency. But if the record exists in more than 4 sites then we
don’t gain any better performance. Again if the record exists in all sites then this
global index causes some overhead because without GI the record is searched in local
sites only, but now the record will be searched in global index file as well as in all
local sites where it exists. That is if a record exists in less than 50% of total sites then
we gain efficiency by the use of GI. So it is a matter of further inquiry how the
records are distributed all over the sites.
We have simulated a program, which assumes 8 sites and 100,000 different no of
individual records according to the indexed attributes. Each site contains 50,000
records randomly. So we see 400,000 no of actual records are distributed all over 8
sites. That is, one individual record can exist in at most 4 sites i.e. 50% of total no of
sites. In that condition, we examined 10,000 individual point queries randomly. It
shows that, before implementing GI, it needs 8*N no of comparison if there are N no
of records in a bucket, but after implementing GI it needs 7*N no of comparison.
Thus we see our performance gain equivalents to 12.5% in terms of comparison.
Again after implementation of GI the cost of network overhead reduces to 50% than
before.
The above result is true when the probability of record distribution is 0.5 among
the sites. If the probability is reduced then the performance gain increases linearly.
2.1.1.1.1 References
[1] “A Distributed Weighted Centroid-based Indexing System, in: Proceedings of the 8th European Networking Conference (JENC8)”, 1997. M. Rio. J. Macedo, V. Freitas, http://www.international. conf.jene8papaers322.ps [2] “An algorithm for the organization of information”, G. M. Adel’son-Vel’skii and E. M. Landis, Doklady Academia Nauk SSSR, vol, 146, no. 2, pp. 263-266, 1962. English translation in Soviet Mathematics, vol.3, no. 5, pp. 1259-1263, 1962. [3] “Architecture of the Whois++ Index Service, RFC1835”, August 1995, P. Deutsch, R. Schoultz, P. Faltstrom, C. Weider, ftp: ftp.ripe.net_rfe_rfe1835.txt [4] “Architecture of the Whois++ Index Service, RFC1913”, February 1996, C. Weider, J. Fullton, S. Spero, ftp://ftp.ripe. net/rfe1913.txt [5] “CIP Index Object Format for SOIF Objects (RFC draft version 2) April 1997”, M. Bowman, D, Hardy, M. Schwartz, D. Wessels, ftp://ftp.ietf.org/internet-drafts/draft-ietf-find-eip-soif-02.txt [6] “Complexity of the Common Indexing Protocol”P. Panotzki, September 1996, http://www.bunyip.com/reaserchpapers1996eip/eip. html [7] ”Database System Concepts”, Abraham Silberschatz, Henry F. Korth, S. Sudarshan, Third Edition,The McGraw Hill Companies, Inc, 1997. [8] “Distributed databases Principals & Systems”, Stefano Ceri, Ginseppe Pelagatti, McGrawHill Book Company, 1984. [9] “Files and Databases: An Introduction”, Peter D. Smith, G. Michael Barnes, Addision-Wesley Publishing Company. [10] “Free Harvest Web Indexing Software Development”, http://www.trdis.ed.ac.uk/harvest [11] “InfoSeek Distributed Search Pattern, 1997”, http:// software/infseek.com/patents/dist.search [12] “Lightweight Directory Access Protocol (v3), RFC2251, December 1997” , M. Wahl, T. Howes, S. Kille, ftp://ftp.ripe.net/ rfc/rfc225.txt [13] “ORACLE 8: A Beginner’s Guide”, Michael Abbey, Michael J. Corey, Tata McGraw -Hill Publishing Company Limited, 1997. [14] “ORACLE The Complete Reference Third Edition” George Koch, Kevin Loney, Osborne McGraw -Hill, 1995. [15] “Organization and maintenance of a large ordered indexes”, R. Bayer, E. McCreight Acta Information, vol 1. No.3 pp, 173-189, 1972. [16] “Proceedings of the TERENA Networking Conference 1998”, P> Valkenburg, D. Beckett, M. Hamilton, S. Wilkinson, Standards in the CHIC-Pilot Distributed Indexing Architecture, in: Computer Networks and ISDN Systems special issue http://www.terena.nl/ libr/tech/ehie-fr.html
[17] “Special Edition Using Microsoft SQL Server 6.5“ Stephen Wynkoop, Prentiee-Hall of India Private Limited, 19998. [18] “Stanford Protocol Proposal fo Internet Search and Retrieval, January 1997”, L. Gravano, K. Chang, H. Garcia -Molina, C, Lagoze, A. Paepcke, http://www.db.stanford.edu/~gravano/ staarts.html [19] “TERNA Task Force on Cooperative Hierarchical Indexing Coordination (TF-CHIC)”, http://www.terena-nl/task/eits.html [20] “The AltaVista Search Service” , http://www.altavista. digital.com [21] “The Architecture of the Common Index Protocol (CIP)” J. Allen, M. Mealling,(RFC draft version 1), 1997, ftp://ftp.ietf.org/internet-drafts/draft-ietf-find-eip-arch-01.txt [22] “The Art of Computer Proogramming vol.3: Sorting and Searching”, D.E. Knuth, Addision-Wesley, Reading Mass, 1973. [23] “The InfoSeek Search Service”, http://www.infoseek.com [24] “W3C’s Distributed Indexing/Searching workshop”, May 1996, http://www.w3.org/Search/9605-Indexing-Workshop