sns college of engineering computer … · ogsa basic services (meta-os and domain services). 5....
TRANSCRIPT
SNS COLLEGE OF ENGINEERING
COMPUTER SCIENCE AND ENGINEERING
CS6703 GRID AND CLOUD COMPUTING
IV YEAR/ VII SEMESTER
2 MARK QUESTIONS WITH ANSWERS
UNIT I: INTRODUCTION
1. What are the advantages of Distributed Computing? The distributed computing environment provides many significant advantages compared
to a traditional standalone application. The following are some of those key advantages: Higher performance. Collaboration. Higher reliability and availability Scalability Extensibility Reuse
2. Define grid computing. Grid concept is defined as control sharing of resources and problem solving in dynamic,
multi institutional virtual organization. Grid computing is a open standard. It has well defined policies and conditions to solving the problem.
3. What are the grid applications? Scheduler Resource broker Grid portals Load Balancing Integrated solutions
4. What is meant by grid infrastructure?
Grid infrastructure is a complex combination of a number of capabilities and resources identified for the specific problem and environment being addressed. It forms the core foundations for successful grid applications. 5. What is the role of a grid portal?
Grid portal provide capabilities for Grid Computing resource authentication, remote resource access, scheduling capabilities, and monitoring status information. 6. What are the available Grid topologies?
Intragrid Local grid within an organisation
Trust based on personal contracts Extragrid
Resources of a consortium of organisations connected through a (Virtual) Private Network
Trust based on Business to Business contracts Intergrid Global sharing of resources through the internet Trust based on certification
7. What are the business benefits in Grid Computing? Acceleration of implementation time frames in order to intersect with the anticipated
business end results. Improved productivity and collaboration of virtual organizations and respective
computing and data resources. Allowing widely dispersed departments and business to create virtual organizations to
share data and resources. 8. What is the role of the grid computing organization? Organizations developing grid standards and best practices guidelines. Organizations developing grid computing toolkits, frameworks and middleware solutions. Organizations building and using grid-based solutions to solve their computing, data, and
network requirements. Organizations working to adopt grid concepts into commercial products, via utility
computing and business on demand computing. 9. What are the business areas needs in Grid computing? Life Sciences Financial services Higher Education Engineering Services Government Collaborative games
10. List out the Grid Applications: Application partitioning that involves breaking the problem into discrete pieces Discovery and scheduling of tasks and workflow Data communications distributing the problem data where and when it is required Provisioning and distributing application codes to specific system nodes Autonomic features such as self-configuration, self-optimization, self-recovery and Self-
management 11. List out the grid portal capabilities: Querying databases or LDAP servers for resource-specific information
File transfer facilities such as file upload, down load, integration with custom software, and so on
Manage job through job status feedbacks Allocate the resources for the execution of specific tasks. Security management Provide personalized solutions
12. What are the areas are difficult to implement in Grid Computing Infrastructure?
A Grid computing infrastructure component must address several potentially complicated areas in many stages of the implementation. These areas are Security Resource management Information services Data management
13. Give the different layers of grid architecture: Fabric Layer: Interface to local resources Connectivity Layer: Manages Communications Collective Layer: Coordinating Multiple Resources Application Layer: User-Defined Application
14. Define Cloud computing with example. Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
For example, Google hosts a cloud that consists of both smallish PCs and larger servers. Google’s cloud is a private one (that is, Google owns it) that is publicly accessible (by Google’s users). 15. Define SOA.
Service oriented architecture is intended to define loosely coupled and interoperable services /applications, and to define process for integrating these interoperable components. In SOA, the system is decomposed into a collection of network-connected components.
Applications and resources within a SOA shouldn’t be built as a tightly coupled monolithic model. Rather these applications are composed dynamically from the deployed and available services in the network. 16. What are the elements of Grid Computing?
Grid computing combines elements such as distributed computing, high-performance computing and disposable computing depending on the application of the technology and the scale of operation.
• Categorized into three views: a functional view,
a physical view, and a service view
17. What are the components of Functional view? Grid portal Security Broker Scheduler Data management Job and resource management Resources
18. What are the components of Physical view?
A grid is a collection of networks, processors, storage, and other resources Networks Computation Storage Scientific Instruments Software and Licenses
19. What is a Service View?
In a Service Oriented Architecture environment, applications are assembled as a collection of services, each of which represents separate and discrete functions or features. As business needs change, services can be added, deleted, or updated as needed, to evolve as the business needs it 20. What are the types of Grid?
• Computational grid This grid is used to allocate resources specifically for computing power. In this situation, most of the processors are high-performance servers.
• Scavenging grid This grid is used to “locate processors–cycles”: grid nodes are exploited for available machine cycles and other resources. In a “scavenging” grid environment, any processor that becomes idle reports its idle status to the grid management node.
• Data grid This grid is used for housing and providing access to data across multiple organizations. 21. Define Grid Architecture
The architecture of a grid system is often described in terms of “layers”, each pro- viding a specific function .Higher layers are user-centric, whereas the lower layers are hardware-centric Network Layer Resource Layer Middleware Layer Application Layer
22. Define functionality of Network Layer and Resource Layer Network layer:
It is the bottom layer which assures the connectivity for the resources in the grid. Resource layer:
It is made up of actual resources that are part of the grid, such as computers, storage systems, electronic data catalogues, and even sensors such as telescopes or other instruments, which can be connected directly to the network 23. Define functionality of Middleware Layer and Application Layer Middleware layer:
It provides the tools that enable various elements (servers, storage, networks, etc.) to participate in a unified grid environment. Application layer:
It includes different user applications (science, engineering, business, financial), portal and development toolkits-supporting applications 24. Define Grid portal in Functional view.
Portal provides the user with an interface to launch applications. The applications make transparent the use of resources and services provided by the grid. With this arrangement, the user perceives the grid as a virtual computing resource.
25. Define Broker function in Functional view.
The grid system needs to identify appropriate and available resources that can be used, based on the application provided by the user. This task is carried out by a broker function.
The broker functionality provides information about the available resources on the grid and the working status of these resources. 26. Define Scheduler function in Functional view.
After available resources have been identified, the follow-on step is to schedule the individual jobs to run on these resources. Schedulers are designed to dynamically react to grid load. They accomplish this by utilizing measurement information relating to the current utilization of processors to determine which ones are available before submitting a job.
• round-robin fashion • job priority (queue) mechanism
27. Define Data Management in Functional view.
A data management functional block usually exists in a grid environment. There typically needs to be a reliable (and secure) method for moving files and data to various nodes within the grid.
28. Define Job Management and Resource Management in Functional view.
The job management and resource management function provides the services to actually launch a job on a particular resource, to check the job’s status, and to retrieve the results when the job is complete. This information is used by the scheduler to decide where grid jobs should be assigned.
This functionality is also known as the grid resource allocation manager (GRAM). 29. What are the Main characteristics of Grids? Large scale Geographical distribution Heterogeneity Resource sharing and coordination Multiple administrations Accessibility attributes
30. Define Computer cluster.
A computer cluster consists of a set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software
The components of a cluster are usually connected to each other through fast local area networks , with each node (computer used as a server) running its own instance of an operating system UNIT II: GRID SERVICES 1. Define OGSA?
Open Grid Services Architecture (OGSA) is a set of standards defining the way in which information is shared among diverse components of large, heterogeneous grid systems. In this context, a grid system is a scalable wide area network (WAN) that supports resource sharing and distribution. OGSA is a trademark of the Open Grid Forum.
2. What are the major goals of OGSA? Identify the use cases that can drive the OGSA platform components. Identify and define the core ODSA platform components. Define hosting and platform-specific bindings. Define resources models and resource profiles with interoperable solutions.
3. What are the Motivations of OGSA? The Open Grid Services Architecture (OGSA) has been proposed as an enabling infrastructure for systems and applications that require the integration and management of services within distributed, heterogeneous, dynamic “virtual organizations” whether within industry, e- science, or e-business. Building on Web services and Grid technologies, OGSA proposes to define a core Grid service semantics 4. What are the layers of OGSA architectural organization? Native platform services and transport mechanism. OGSA hosting environment. OGSA transport security. OGSA infrastructure (OGSI). OGSA basic services (meta-OS and domain services).
5. Define OGSA Platform The core set of interfaces, behaviors, profiles, models, and bindings that address these issues form what we term the OGSA Platform. The OGSA Platform encompasses not only broadly applicable service definitions but also models for commonly used components. 6. What are the core requirements for describing web services based on OGSI? The ability to describe the interface inheritance. The ability to describe additional information elements (state data/attributes/properties)
with the interface definitions. 7. What are the SDE attributes of OGSI schema type for service data? The new OGSI schema type contains seven predefined attributes, including name, type, minOccurs, maxOccurs, modifiable, mutability, and nilable. These are standard XSD types with the exception of the “mutability” attribute. This is further defined by OGSI as an enumerated type, with the values of “static”, “constant”, “extendable” and “mutable”. 8. What are the lifetime properties of OGSI specification? This time from which the contents of this element are valid (ogsi: goodFrom). This time until which the contents of this element are valid (ogsi:goodUntil). This time until which this element itself is available (ogsi: availableUntil).
9. What is soft-state lifetime management?
The soft-state lifetime management approach is a recommended method in the grid service life-cycle management process. Every grid service has a terminated time set by the service creator. This soft-state lifecycle is controlled by appropriate security and policy decisions of the service and the service has the authority to control this behavior.
10. What are Grid service Interfaces? The grid service interfaces and their associated behaviors are described by the OGSI specification. This OGSI interfaces are classified into three sets of interfaces based upon their functionality. They are OGSI core Notification Service groups.
11. What are the rules for handling the services of the set operation on service data? The rules are as follows: The service Data must be modifiable; the SDE-modifiable attribute must be “true”. The service Data mutability attribute should not be “static” or “constant”. If the service Data mutability attribute is “extendable” the set operation must append the
new SDE values to the existing SDE values. If the service Data mutability attribute is “mutable” the set operation must replace the
existing SDE values with the new SDE values. The SDE values, “append” and “replace” must adhere to the minOccurs and maxOccurs
attributes on SDE values. 12. What is “handle resolver”? Handle resolving is a standard mechanism to resolve a GSH into a GSR. This is an optional feature based on the Handler solver port Type. A grid service instance that implements the Handler solver port Type is called a “handle resolver”. 13. Explain about Membership Content Rule. Deriving a service from the Service Group port Type and utilizing the “Membership Content Rule” service data for the classification mechanisms can create a grouping concept similar to a registry. This “rule” service data is used to restrict the membership of a grid service in the group. 14. What are the two types of clients in Grid Service? Static: These kinds of clients have plurality of a priori knowledge on the runtime binding
information. These are faster but less flexible in operations.
Dynamic: These types of clients are flexible and they are not bound to any predefined artifacts. These are highly flexible but less efficiencies.
15. What is service domain? The OGSA service domain architecture proposes a high level abstraction model to describe the common behaviors, attributes, operations and interfaces to allow a collection of services to function as a single unit.
16. What are the functionalities provided by the service domain components? Service registration and collection Service routing and selection Service interoperation and transformation Flexible service composition Automatic service orchestration
17. What are the various levels of Policy Abstraction? Business Level Domain Level Device Level
18. What is RFT? The reliable file transfer service (RFT) is an OGSA based service that provides interfaces for controlling and monitoring third party file transfers using the GridFTP servers. The client controlling the transfer is hosted inside of a grid service. 19. Define OGSI OGSI defines essential building blocks for distributed systems, including standard interfaces and associated behaviors for describing and discovering service attributes, creating service instances, managing service lifetime, and subscribing to and delivering notifications. However, it certainly does not define all elements that arise when creating large-scale systems. 20. Which technologies do OGSI based on?
OGSA is based primarily on the technologies of Web Service Description Language (WSDL) and Simple Object Access Protocol (SOAP), an XML-based protocol for passing messages between systems over the Internet. It is service- oriented because it works as a series of loosely coupled, interacting services that use industry-accepted Web services standards. 21. What OGSA does? OGSA builds on concepts and technologies from both the grid and Web services
communities, defining a uniform vocabulary and grammar for grid services. The Open Grid Services Architecture (OGSA) defines mechanisms for creating,
managing and exchanging information among entities called grid services. 22. What are the basic categories of Data Management?
Two Basic Categories of Data Management are Data Movement
Secure Robust Efficient Third party movement
Data Replication One or more copies or replicas
Survive loss Easy availability
23. List some Advantages of Grid Computing Can solve larger, more complex problems in a shorter time Easier to collaborate with other organizations Make better use of existing hardware
24. List some Disadvantages of Grid Computing Grid software and standards are still evolving Learning curve to get started Non-interactive job submission
25. What are the available OGSA services? Infrastructure services Execution management services Data services Resource management services Security services Self-management services Information services
26. Define Execution Management Services. It Concerned with problems of instantiating and managing, to complete units of work It allows problems in executing units of work
Finding execution candidate locations Selecting execution location Preparing for execution Initiating the execution Managing the execution
It also includes EMS services such as resources, job management and resource selection services
27. Define Data Services. It allow Execution Management Service to access the data remotely and store in local
machine It operate with generic data Management of, access to and update of data resources, along with the transfer of data
between resources It supports
Remote access Staging Replication
Federation Derivation Metadata
28. Define Resource Management Services Management of resources in a grid Three types of management
Management of the physical and logical resources Management of the OGSA grid resources exposed through service interface Management of the OGSA grid infrastructure, exposed through its management
interfaces 29. Define Self Management Service. operating an IT infrastructure System components are self-configuring, self-healing and self-optimizing
Self-configuring mechanism Self-healing mechanism Self-optimizing mechanism
30. Define Information Services. Ability to access and manipulate information about applications, resources and services
in the grid environment Information – dynamic data or events Demand placed on the source of information and QoS requirement
Message delivery Logging Monitoring
UNIT III: VIRTUALIZATION 1. What are the Cloud deployment models?
Public Cloud infrastructure is available to the general public, owned by organization
selling cloud services Private
Cloud infrastructure for single org only, may be managed by the organization or a 3rd party, on or off premise
Community Cloud infrastructure shared by several organizations that have shared
concerns, managed by org or 3rd party Hybrid
Mixture of public cloud and private cloud. Critical activities are performed using private cloud and non- Critical activities are performed using public cloud.
2. What are the properties of Cloud Computing?
There are six key properties of cloud computing user-centric task-centric powerful accessible intelligent programmable
3. What is the working principle of Cloud Computing? The cloud is a collection of computers and servers that are publicly accessible via the Internet. This hardware is typically owned and operated by a third party on a consolidated basis in one or more data center locations. The machines can run any combination of operating systems. 4. Draw the architecture of Cloud
5. Define Cloud services with example. Any web-based application or service offered via cloud computing is called a cloud service. Cloud services can include anything from calendar and contact applications to word processing and presentations. 6. What are the advantages of cloud services? If the user’s PC crashes host application and document both remain unaffected in the
cloud. An individual user can access applications and documents from any location on any PC. Because documents are hosted in the cloud, multiple users can collaborate on the same
document in real time, using any available Internet connection. Documents are not machine-centric
7. Why is Cloud Computing important?
There are many implications of cloud technology, for both developers and end users.
For developers, cloud computing provides increased amounts of storage and processing power to run the applications they develop. Cloud computing also enables new ways to access information, process and analyze data, and connect people and resources from any location anywhere in the world.
For users, documents hosted in the cloud always exist, no matter what happens to the user’s machine. Users from around the world can collaborate on the same documents, applications, and projects, in real time. And cloud computing does all this at lower costs, because the cloud enables more efficient sharing of resources than does traditional network computing.
8. What are the advantages of Cloud Computing?
Lower-Cost Computers for Users Improved Performance Lower IT Infrastructure Costs Fewer Maintenance Issues Lower Software Costs Instant Software Updates Increased Computing Power
9. What are the disadvantages of Cloud Computing?
Requires a Constant Internet Connection Doesn’t Work Well with Low-Speed Connections Can Be Slow Features Might Be Limited Stored Data Might Not Be Secure If the Cloud Loses Your Data, You’re Screwed
10. Who get benefits from Cloud Computing?
Collaborators Road Warriors Cost-Conscious Users Cost-Conscious IT Departments Users with Increasing Needs
11. Who shouldn’t be using Cloud Computing?
The Internet-Impaired Offline Workers The Security Conscious Anyone Married to Existing Applications
12. List the advantages and disadvantages of cloud service deployment. Advantages
Economy of scale
Offer better, cheaper, and more reliable applications Utilization of the full resources Less up-front investment Rapid provisioning Automatic scaling
Disadvantages Security Need Redundancy Tool No physical backup
13. What are the types of Cloud service development?
Software as a Service Platform as a Service Infrastructure as a service
14. List the companies who offer cloud service development?
Amazon Google App Engine IBM Salesforce.com
15. What Is Virtualization And Cloud Computing?
Virtualization is an emerging IT paradigm that separates computing functions and technology implementations from physical hardware.
Cloud computing, for example, is the virtualization of computer programs through an internet connection rather than installing applications on every office computer. 16. How Virtualization Achieves Optimal Data Storage
Virtual data storage operates similar to a bank:
A client would bring their data to a cloud computing firm who would store it. The technology company would take that data, optimize it, and merge it with the client’s
existing data storage. When the client needs to get their data, a single internet access point delivers the required
email or file. To streamline the process, periodic data archiving can automatically occur to keep the
stored information current while creating a turn-key operation. 17. What are the Benefits of using Virtual Machines? Instant provisioning - fast scalability Live Migration is possible Load balancing and consolidation in a Data Center is possible. Low downtime for maintenance Security and fault isolation
18. What is the importance of Virtualization in Cloud Computing? Cloud can exist without Virtualization, although it will be difficult and inefficient. Cloud makes notion of "Pay for what you use" and "infinite availability- use as much you
want". These notions are practical only if we have
Lot of flexibility. Efficiency in the back-end.
This efficiency is readily available in Virtualized Environments and Machines.
19. List out disadvantages of Virtualization Virtualization may not work well for: Resource-intensive applications VMs may have RAM/CPU limitations Performance testing Hardware compatibility testing Specific hardware requirements
Some hardware architectures or features are impossible to virtualize such as: Certain registers or state not exposed Clocks, time, and real-time behavior
20. What are the available Virtualization Techniques? Full virtualization using Binary Translation. OS Assisted Virtualization or Paravirtualization. Hardware Assisted Virtualization.
21. What is Privilege Rings? CPUs provide a range of protection levels also known as rings in which code can execute. Ring 0 has the highest level privilege.
22. Define Full virtualization
Almost complete simulation of the actual hardware to allow software, which typically consists of a guest operating system, to run unmodified.
23. Define Para virtualization.
A hardware environment is not simulated; however, the guest programs are executed in their own isolated domains, as if they are running on a separate system. Guest programs need to be specifically modified to run in this environment.
24. Define Hardware Assisted Virtualization It is a way of improving the efficiency of hardware virtualization. It involves employing
specially designed CPUs and hardware components that help improve the performance of a guest environment.
25. Define Hypervisor.
In virtualization, the host machine is the actual machine on which the virtualization takes place, and the guest machine is the virtual machine. The software or firmware that creates a virtual machine on the host hardware is called a hypervisor or Virtual Machine Manager which is the low-level program that allows multiple operating systems to run concurrently on a single host computer
Two types of Hypervisors are Type 1 Type 2
26. Define Type 1 Hypervisor. A Type-1 hypervisor interacts directly with hardware that is being virtualized.
It is completely independent from the operating system. Boots before the operating system (OS).
They are often referred to as a "native" or "bare metal" or "embedded" hypervisors in vendor literature.
27. Define Type 2 Hypervisor. It relies heavily on the operating system. It cannot boot until the operating system is already up and running. If operating system crashes, all end-users are affected. Since Type-2 hypervisors depend on an OS, they are not in full control of the end user's
machine. 28. What are the virtualization challenges available? CPU virtualization Memory virtualization I/O virtualization
29. What are the drawbacks of Full virtualization? Hardware emulation comes with a performance price In traditional x86 architectures, OS kernels expect to run privileged code in Ring 0
However, because Ring 0 is controlled by the host OS, VMs are forced to execute at Ring 1/3, which requires the VMM to trap and emulate instructions
Due to these performance limitations, par virtualization and hardware-assisted virtualization were developed
30. What are the pros and Cons of Hardware-assisted virtualization • Pros
It allows to run unmodified OS (so legacy OS can be run without problems) • Cons
Speed and Flexibility An unmodified OS does not know it is running in a virtualized environment and
so, it can’t take advantage of any of the virtualization features It can be resolved using paravirtualization partially
UNIT IV: PROGRAMMING MODEL
1. List out some of the Grid Computing Toolkits:
Globus Legion Condor and Condor-G Nimrod
2. Define globus Toolkit
The Globus Toolkit (GT) has been developed since the late 1990s to support the development of service-oriented distributed computing applications and infrastructures.
a set of libraries and programs that address common problems that occur when building distributed system services and applications.
3. Discuss about Globus Project
The Globus Project is a joint effort on the part of researchers and developers from aroundthe world that are focused on the concept of Grid computing. It's organized around fourmain activities:
Research Software tools Testbeds Applications
4. What specific problem is the Globus Toolkit designed to address?
Ultimately, the Globus Toolkit is designed to enable the creation and maintenance of Virtual Organizations . 5. What are all the protocols used in GT?
Web service protocols WSDL, SOAP WS Addressing, WSRF, WSN WS Security, SAML, XACML WS-Interoperability profile
Non Web service protocols Standards-based, such as GridFTP Custom
6. What are the Domain Areas in GT? Core runtime: Infrastructure for building new services Security: Apply uniform policy across distinct systems Execution management: Provision, deploy, & manage services Data management: Discover, transfer, & access large data Monitoring: Discover & monitor dynamic services
7. List out the characteristics of Virtual organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals Dynamic Fault tolerant What are the goals of GT4 Usability, reliability, scalability Consistency with latest standards (WS-*, WSRF, WS-N, etc.) and Apache platform New components, platforms, languages
8. Define Globus Security It provides Control access to shared services It Support multi-user collaborations It also allow users and application communities to set up dynamic trust domains
9. What is a Hadoop?
Hadoop is an Apache open source framework written in java that allows distributed processing of large datasets across clusters of computers using simple programming models.
A Hadoop frame-worked application works in an environment that provides distributed storage and computation across clusters of computers.
Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage. 10. Is Hadoop open source?
The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. 11. What are the elements of Hadoop Architecture? Hadoop Common Hadoop YARN Hadoop Distributed File System (HDFS) Hadoop MapReduce
12. What is the HDFS?
Hadoop Distributed File System (HDFS) is a Java-based file system that provides scalable and reliable data storage that is designed to span large clusters of commodity servers.
A distributed file system that provides high-throughput access to application data. HDFS uses a master/slave architecture where master consists of a single NameNode that
manages the file system metadata and one or more slave DataNodes that store the actual data 13. What is Mapreduce?
Hadoop MapReduce is a software framework for easily writing applications which process big amounts of data in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
The term MapReduce actually refers to the following two different tasks that Hadoop programs perform:
The Map Task The Reduce Task
14. Define Map Task and Reduce Task
The Map Task: This is the first task, which takes input data and converts it into a set of data, where individual elements are broken down into tuples (key/value pairs).
The Reduce Task: This task takes the output from a map task as input and combines those data tuples into a smaller set of tuples. The reduce task is always performed after the map task.
15. List out advantages of Hadoop Hadoop framework allows the user to quickly write and test distributed systems. It is
efficient and it automatic distributes the data and work across the machines and in turn, utilizes the underlying parallelism of the CPU cores.
Servers can be added or removed from the cluster dynamically and Hadoop continues to operate without interruption.
Another big advantage of Hadoop is that apart from being open source, it is compatible on all the platforms since it is Java based.
16. What are the available Hadoop Operation Modes? Local/Standalone Mode Pseudo Distributed Mode Fully Distributed Mode
17. What are the elements of HDFS Architecture? NameNode DataNode Block
18. Define Namenode The namenode is the commodity hardware that contains the GNU/Linux operating
system and the namenode software. It is software that can be run on commodity hardware. The system having the namenode acts as the master server and it does the following tasks:
It also manages the file system namespace. It regulates client’s access to files. It also executes file system operations such as renaming, closing, and opening files and
directories.
19. Define Datanode The datanode is a commodity hardware having the GNU/Linux operating system and
datanode software. For every node (Commodity hardware/System) in a cluster, there will be a datanode. These nodes manage the data storage of their system.
Datanodes perform read-write operations on the file systems, as per client request. They also perform operations such as block creation, deletion, and replication according
to the instructions of the namenode.
20. Define Block Generally the user data is stored in the files of HDFS. The file in a file system will be
divided into one or more segments and/or stored in individual data nodes. These file segments are called as blocks. In other words, the minimum amount of data
that HDFS can read or write is called a Block. The default block size is 64MB, but it can be increased as per the need to change in
HDFS configuration. 21. List the features of HDFS It is suitable for the distributed storage and processing. Hadoop provides a command interface to interact with HDFS. The built-in servers of namenode and datanode help users to easily check the status of
cluster. Streaming access to file system data. HDFS provides file permissions and authentication.
22. What are the Goals of HDFS? Fault detection and recovery: Since HDFS includes a large number of commodity
hardware, failure of components is frequent. Therefore HDFS should have mechanisms for quick and automatic fault detection and recovery.
Huge datasets: HDFS should have hundreds of nodes per cluster to manage the applications having huge datasets.
Hardware at data: A requested task can be done efficiently, when the computation takes place near the data. Especially where huge datasets are involved, it reduces the network traffic and increases the throughput.
23. Replication causes data redundancy, then why is it pursued in HDFS?
HDFS works with commodity hardware (systems with average configurations) that has high chances of getting crashed any time. Thus, to make the entire system highly fault-tolerant, HDFS replicates and stores data in different places.
Any data on HDFS gets stored at least 3 different locations. So, even if one of them is corrupted and the other is unavailable for some time for any reason, then data can be accessed from the third one.
Hence, there is no chance of losing the data. This replication factor helps us to attain the feature of Hadoop called Fault Tolerant. 24. Since the data is replicated thrice in HDFS, does it mean that any calculation done on one node will also be replicated on the other two?
No, calculations will be done only on the original data. The master node will know which node exactly has that particular data. In case, if one of the nodes is not responding, it is assumed to be failed. Only then, the required calculation will be done on the second replica. 25. What is a heartbeat in HDFS?
A heartbeat is a signal indicating that it is alive. A datanode sends heartbeat to Namenode and task tracker will send its heart beat to job tracker. If the Namenode or job tracker does not
receive heart beat then they will decide that there is some problem in datanode or task tracker is unable to perform the assigned task. 26. Explain how do ‘map’ and ‘reduce’ works.
Namenode takes the input and divide it into parts and assign them to data nodes. These datanodes process the tasks assigned to them and make a key-value pair and returns the intermediate output to the Reducer. The reducer collects this key value pairs of all the datanodes and combines them and generates the final output. 27. Why ‘Reading‘ is done in parallel and ‘Writing‘ is not in HDFS?
Through mapreduce program the file can be read by splitting its blocks when reading. But while writing as the incoming values are not yet known to the system mapreduce cannot be applied and no parallel writing is possible. 28. What are Problems with small files and HDFS? HDFS is not good at handling large number of small files. Because every file, directory and block in HDFS is represented as an object in the namenode’s memory, each of which occupies approx 150 bytes, So 10 million files, each using a block, would use about 3 gigabytes of memory. when we go for a billion files the memory requirement in namenode cannot be met. 29. Why is Checkpointing Important in Hadoop? As more and more files are added the namenode creates large edit logs. Which can
substantially delay NameNode startup as the NameNode reapplies all the edits. Checkpointing is a process that takes an fsimage and edit log and compacts them into a
new fsimage. This way, instead of replaying a potentially unbounded edit log, the NameNode can load the final in-memory state directly from the fsimage.
This is a far more efficient operation and reduces NameNode startup time. 30. What is a Combiner? The Combiner is a ‘mini-reduce’ process which operates only on data generated by a
mapper. The Combiner will receive as input all data emitted by the Mapper instances on a given
node. The output from the Combiner is then sent to the Reducers, instead of the output from the
Mappers
UNIT V: SECURITY 1. Define Grid security infrastructure The Grid Security Infrastructure (GSI) is a set of tools, libraries and protocols used in
Globus to allow users and applications to securely access resources. Accessing is based on a public key infrastructure, with certificate authorities and X509
certificates Uses Secure Socket Layer for authentication and message protection
2. Define Secure Socket Layer (SSL) Protocol above a standard TCP/IP socket to provide security in the forms of: Authentication Message protection Confidentiality Integrity
3. Explain about GSI Credentials
In the GSI system each user has a set of credentials they use to prove their identity on the grid
Consists of a X509 certificate and private key Long-term private key is kept encrypted with a pass phrase
Good for security, inconvenient for repeated usage
4. Explain about GSI: Single Sign-on Single-sign on is important feature for Grid Applications Enables easy coordination of multiple resources User authenticates themselves once, then can perform multiple actions without
reauthentication Can allow processes to act on their behalf To support single sign-on GSI adds the following functionality to SSL: Proxy credentials Credential delegation
5. Explain about GSI: Proxy Credentials
Proxy credentials are short-lived credentials created by user Short term binding of user’s identity to alternate private key Stored unencrypted for easy repeated access Short lifetime in case of theft Enables user to authenticate once then perform multiple actions without
reauthenticating 6. Explain about GSI: Delegation
GSI enables user to create and delegate proxy credentials to processes running on remote resources
Allows remote processes and resources to act on user’s behalf Important for complex applications that need to use Grid resources E.g. jobs that needs to access data storage
7. What are the available cloud models?
Delivery Models SaaS PaaS IaaS
Deployment Models Private cloud Community cloud Public cloud Hybrid cloud
8. What are the problems associated with Cloud Computing? Most security problems stem from:
Loss of control Lack of trust (mechanisms) Multi-tenancy These problems exist mainly in 3rd party management models Self-managed clouds still have security issues, but not related to above
9. What are the essential Characteristics of Cloud?
Resource pooling Provider resources pooled to server multiple clients Rapid Elasticity Ability to quickly scale in/out service Measured service control, optimize services based on metering
10. If cloud computing is so great, why isn’t everyone doing it?
The cloud acts as a big black box, nothing inside the cloud is visible to the clients Clients have no idea or control over what happens inside a cloud Even if the cloud provider is honest, it can have malicious system admins who
can tamper with the VMs and violate confidentiality and integrity Clouds are still subject to traditional data confidentiality, integrity, availability,
and privacy issues, plus some additional attacks 11. Explain about Network level in cloud.
Ensuring confidentiality and integrity of your organization’s data-in-transit to and from your public cloud provider
Ensuring proper access control (authentication, authorization, and auditing) to whatever resources you are using at your public cloud provider
12. Explain about Host Level in cloud.
Both the PaaS and SaaS platforms abstract and hide the host OS from end users Host security responsibilities are transferred to the CSP (Cloud Service Provider) You do not have to worry about protecting hosts However, as a customer, you still own the risk of managing information hosted in
the cloud services.
13. What are the aspects of Data security Data-in-transit Data-at-rest Processing of data, including multi tenancy Data lineage Data provenance
14. What are the services, advantages, examples of Saas with its Service Provider?
Service Models
Services Example Service Providers
Advantage
Saas
(Consume)
Software is offered as
Service and delivered
through a browser
Excel, Web Page, CRM,
ERP
Access, SQL Server
Google Apps
Salesforce.com
Reduce the cost
Centralized control
15. What are the services, advantages, examples of Paas with its Service Provider?
Service Models
Services Example Service Providers
Advantage
Paas
(build on it)
Enables developers to
write applications
without installing any
tools in local system
but run on the cloud.
Scripting
Coding
Coding and integration
App Engine
Azure
Engine Yard
Force.com
Scalability,
Reliability and security
Pay-per-use
16. What are the problems faced in GSI infrastructure In GSI thousands of users ,thousands of Certs, many of CAs (with different policies) are
present Grid-wide user group and roles are needed
No grid-wide logging or auditing Need for anonymous users Protocol are needed to access personal credential for OCR
17. What are the services, advantages, examples of Iaas with its Service Provider?
Service Models
Services Example Service Providers
Advantage
Iaas (Migrate to it)
Computing
infrastructure is rented
to the user
Infrastructure
Scalability &
Availability
Amazon EC2,S3
Go Grid
Linode
Rackspace
Scalability
Pay as you go
Best-of-breed
Technology and resources
18. What are the classifications of security issues in cloud computing?
The security issues in cloud computing can be categorized into the following three broad classes: Traditional security concerns Availability issues Third party data control-related issues
19. Define Traditional Security Issues
These security issues involve computer and network intrusions or attacks that will be made possible or atleast easier by moving to the cloud.
Cloud providers respond to these concerns by arguing that their security measures and processes are more mature and tested than those of the average company.
It may be easier to enforce security via contracts with online services providers than via internal controls. 20. Define Third Party Data Control
The legal implications of data and applications being held by a third party are complex and not well understood.
There is also a potential lack of control and transparency when a third party holds the data.
Part of the hype of cloud computing is that the cloud can be implementation-independent, but in reality, regulatory compliance requires transparency into the cloud. 21. What are the Emerging Cloud Security Threats? Side channel attacks Denial of service attacks Social networking attacks Mobile device attacks Insider and organized crime threat Cheap data and data analysis
Cost-effective defense of availability Mash-up authorization
22. What are the types of attackers in cloud computing?
Many of the security threats and challenges in cloud computing will be familiar to organizations managing in house infrastructure and those involved in traditional outsourcing models. Each of the cloud computing service delivery models’ threats result from the attackers that can be divided into two groups Internal attackers External attackers
23. What are the characteristics of internal attackers? An internal attacker has the following characteristics: Is employed by the cloud service provider, customer or other third party provider
organization supporting the operation of a cloud service May have existing authorized access to cloud services, customer data or supporting
infrastructure and applications, depending on their organizational role Uses existing privileges to gain further access or support third parties in executing attacks
against the confidentiality integrity and availability of information within the cloud service
24. What are the characteristics of External attackers? An external attacker has the following characteristics: Is not employed by the cloud service provider, customer or other third party provider
organization supporting the operation of a cloud service Has no authorized access to cloud services, customer data or supporting infrastructure
and applications Exploits technical, operational, process and social engineering vulnerabilities to attack a
cloud service provider, customer or third party supporting organization to gain further access to propagate attacks against the confidentiality, integrity and availability of information within the cloud service.
25. How attackers are categorized in cloud environment?
In the cloud environment, attackers can be categorized into four types: random weak strong substantial
26. List Security risks in Cloud computing Privileged user access Data location and segregation Data disposal e-investigations and Protective monitoring
Assuring cloud security 27. What is meant by Random and weak attacker?
Random attacker is the most common type of attacker uses simple tools and techniques. The attacker may randomly scan the Internet trying to find vulnerable components. They will deploy well known tools or techniques that should be easily detected.
Weak attackers are Semi-skilled attackers targeting specific servers/cloud providers by customizing existing publicly available tools or specific targets. Their methods are more advanced as they attempt to customize their attacks using available exploit tools. 28. What is meant by strong and substantial attacker?
Strong attacker is Organized, well-financed and skilled groups of attackers with an internal hierarchy specializing in targeting particular applications and users of the cloud. Generally this group will be an organized crime group specializing in large scale attacks.
Substantial attacker is motivated, strong attackers not easily detected by the organizations they attack, or even by the relevant law enforcement and investigative organizations specializing in e-Crime or cyber security. 29. Define IAM.
Authentication Verifying the identity of a user, system or service
Authorization Privileges that a user or system or service has after being authenticated (e.g.,
access control) Auditing
Exam what the user, system or service has carried out Check for compliance
30. What are the challenges faced by IAM Provisioning resources to users rapidly to accommodate their changing roles Handle turnover in an organization Disparate dictionaries, identities, access rights Need standards