Features

GWE comes with a wide range of features. The following list just describes the most relevant and important, categorized according to end user functionality, scalability and extensibility and overall product robustness.

Functionality

These are the features the end user will immediately benefit and care about.

  • Simple XML based configuration, to describe user's virtual grid configurations.
  • Simple XML based configuration to provide user's authentication information for GWE to access networked grid resources (such as remote file systems for reading/writing files) on the user behalf.
  • Three generic client applications for each flavor of user: rich interface, interactive console and shell scripts.
  • Powerful language (P2EL ) to describe process parallelization invocations (orders).
  • Order submission (collection of parallelizable processes) to a described virtual grid for parallel execution.
  • Transparent staging and translation of input and output files.
  • Execution management of orders (pause, resume, abort).
  • On demand execution progress monitoring of orders in the form of results of queries submitted to a GWE daemon.
  • Real time execution monitoring of orders in the form of GWE daemon real time events remotely streamed.
  • Transparent administration through silent deployment of the distributed systems across user's grid.
  • Auto discovery of resource managers. Supported out of the box: Local multi-processes (default if no other found), Condor and "qsub" based ( PBS , SGE , Torque , LSF ).
  • System events generation and logging.

Scalability

These are features to be exploited by developers to create their own custom implementations of different GWE components:

  • Rich client API to create custom GWE clients.
  • Abstract job descriptors to create custom programmatically describable collection of parallelizable processes. P2EL is built on top of this framework.
  • Abstract job result parsers to create custom parsers for process results.
  • Parameterizable orders to provide custom data for the processes.
  • Parameterizable orders to provide custom behavior that should be invoked at the different phases a process goes through according to the orderŐs callback signature.
  • Driver management framework to accept new thin drivers to manage different grid related type of resources. These resource types are:
    • File Systems. Out of the box support: Local, HTTP and SFTP file systems.
    • Network Protocols. Out of the box support: Local and SSH.
    • Resource Managers. Out of the box support: Local, Condor and "qsub" based ( PBS , SGE , Torque , LSF ).

Robustness

These are internal, low level features; which are meant to give GWE the robustness level expected from a true industrial strength enterprise system:

  • High performance, secured RPC network backbone built using Java RMI over SSH tunnels used as the transport layer for all GWE's inter-subsystems communications.
  • Internal relational database to provide true data warehousing for the distributed system.
  • Virtual file system where and staged files are stored and cached as well as files meant to be uploaded to a final remote destination.
  • Highly multi threaded system to achieve maximum performance. Most possible wait cycles have been isolated doing this and great care has been taken to guarantee the system is thread safe.
  • Embedded health check services (health status messages, fail over and recovery) for distributed agent proxies.
  • Automatic fail over and recovery from cluster and environment related problems.
  • Automatic fail over and recovery from processes related problems.
  • Other general preventive maintenance services. For example recycling unused files from users virtual file systems to free up hard disk space.