MENU Services

o Supercomputingo Datao Visualisationo Training o User Feedback

DATAProjects using the Pawsey Supercomputing Centre’s resources often produce seemingly unmanageable amounts of data. Not only does this data need to be stored, but scientific computing places new demands on researchers to curate and share their data, as well as to ensure the value, accuracy and longevity of their results.

In addition to providing data storage for the most demanding research projects, our Data staff help researchers to maximise the potential of these vast data sets through a number of support services including:

Identifying and minimising bottlenecks in workflows. Working with researchers to optimise research workflows. Regular training workshops on best-practice methods and

technology.

Services AvailableThe Pawsey Supercomputing Centre currently houses in excess of 40 Petabytes of data storage resources. The Centre also has a new research cloud available to Australian researchers called Nimbus, which provide new facilities for data intensive science, data analytics, machine learning and scientific workflow.

1. Data Storage

Our Data Team also provides expertise to help researchers fully utilise Pawsey resources and improve the management of their research data. Managing data involves storing the physical data sets, adding and publishing metadata to make data sets discoverable and controlling access to the data. If you’re new to such issues then you may wish browse our Data Guides and Help .

PAWSEY SUPERCOMPUTING CENTRE DATA STORESThe Pawsey Supercomputing Centre provides data storage services to the research community, on a merit basis. Access to the Pawsey Supercomputing Centre Data Stores is governed by a Data Storage and Management Policy (DSMP), which was approved by the Pawsey Supercomputing Centre Board. The DSMP is available to all researchers from here .

Access to the Pawsey Supercomputing Centre Data StoresAll data storage allocations are started here . Please note:

All researchers will be asked to provide a project short name which will be associated with the directory provided.

Every project has a primary and a deputy custodian who are responsible for the data – only these custodians can grant access to others.

Applications < 5TB are generally not considered unless there are exceptional circumstances (eg multi-participant high-value data sets.)

Applications > 100TB will be referred to the Supercomputing Centre Data Allocation Committee for review.

The Pawsey Supercomputing Centre will only approve applications where a second copy of the data is available elsewhere (except under exceptional circumstances.)

The only access mechanism for the Pawsey Supercomputing Centre Data Stores is via LiveARC (also known as Mediaflux.)

At this time, due to funding, the Pawsey Supercomputing Centre Data Stores will only provide storage allocations until June 2019.

All researchers using the Data Store must comply with the Pawsey Supercomputing Centre Data Storage and Management Policy.

My application was successful, how do I ingest data to the Pawsey Supercomputing Centre?

The Pawsey Supercomputing Centre’s primary data ingest tool is the Data Portal .

If you’re having trouble, please consult the list of frequently asked questions .

DATA GUIDES AND HELPIntroduction to Research Data Management Guides

The significance of research data and developing methods to manage the ever increasing quantities is a common issue amongst researchers. Reproducibility or the ability for another researcher to replicate analyses to validate theories and produce complementary or entirely new science is essential. Science is also increasingly based on computation; making access to the data and code needed to reproduce results increasingly important. Hence researchers should aim to make their research data FAIR (Findable, Accessible, Interoperable / Intelligible, Reusable).

To assist researchers navigate the bewildering array of information on research data management and make their data ‘FAIR’, the Pawsey Supercomputing Centre has developed some guides on research data management. If you are a researcher from one of our partners, you should refer to the policies, procedures and resources provided by your local institution before starting to implement any data management. Below are links to guides from the four public WA universities (accurate as of 21st January 2015)

CSIRO – https://confluence.csiro.au/display/RDM/Research+Data+Management+Home (CSIRO users only)

Curtin University- http://libguides.library.curtin.edu.au/research-data-management

ECU – http://ecu.au.libguides.com/research_data_management Murdoch University – http://libguides.murdoch.edu.au/RDM UWA – http://www.is.uwa.edu.au/research/research-data-

management-toolkit

The Pawsey Supercomputing Centre research data management guides are for general advice only and researchers should consider the appropriateness in relation to their own research and institutional policies before implementing.  If you have any questions relating to the guides or our data services, please contact: data@pawsey.org.au 

We also acknowledge the support of ANDS (Australian National Data Service) in reviewing documentation and providing some of the content.

A checklist for each topic is provided for your reference.  These are separate to the actual guides which are available to download using the links below.

1. Data Ownership, Legal and Ethical

Is the owner / Principal Investigator of the data clearly identified? Does your data contain confidential or sensitive information? If

so, have you discussed data sharing with the respondents and gained consent from whom you collected the data?

Do you need to anonymise data? (e.g. to remove identifying information or personal data)

If appropriate, have you gained ethics approval from your relevant committee?

Are there any cultural and/or commercial sensitivities to your data?

Have you established who owns the copyright of your data? Is there joint copyright?

Have you created an appropriate license for the data?

1_Data_ownership_legal_ethical_guide

2. Data Documentation

Are you using standardised and consistent procedures to collect, process, check, validate and verify data?

Are the variable names, codes and abbreviations used explained? How will you label and organise data, records and files? Have you used standardised keywords / controlled vocabulary in

your metadata? Has a collection-level metadata record been published to a

relevant discovery portal?

2_Data_documentation_guide

3. Data Storage and Sharing

How much data are you likely to collect during your project? Where will your data be stored during / after the project? What is your backup strategy ? Which data formats will you use? Do your formats and software

enable sharing and long-term validity of data, such as non-proprietary software and software based on open standards?

Is there non-digital data? Where will this data be held? Do you need to securely store personal or sensitive data? If data are held in various places, how will you keep track of

versions? Who has access to which data during and after research? Are

various access arrangements needed?

3_Data_storage_sharing_guide

4. Data Publication and Re-use

Has a permanent identifier (e.g. DOI) been minted for the data? Have you considered the costs associated with making the data

accessible? How long should the data be retained for?

Are there any restrictions on the re-use of data? Is this reflected in the license?

Who will retain custodianship of the data?

4_Data_Publication_Re-use_Guide

Sourced from: University of Essex, UK Data Archive (www.data-archive.ac.uk); eResearch SA Research Data Management checklist – Theresa McGinley (http://www.ersa.edu.au/enewsapril2014/data-management-checklist) 21st January 2015.

CLOUD SERVICESPawsey has launched its new Cloud computing service, Nimbus, and is now accepting applications for the Early Adopter programme.

What is Nimbus?Nimbus is a free new Cloud service, open to anyone. Specifically designed for research applications, Nimbus facilitates large data workflows and computational tasks, and offers a data analytics capability.

On Nimbus, researchers will have:

Fast access to the Pawsey data storage facility High Performance Computing on the OpenStack system; this is

especially beneficial for smaller projects (researchers can continue to use Magnus for larger projects)

A centralised system and toolset for data science applications, such as data analytics and machine learning, with data centric workflows and visualisation

Open access for Australian researchers, including state government researchers.

Dedicated expert support team. A new application process, supporting scientific research. We will have more to talk about as we expand and enhance the

service over the coming year.

Nimbus Early Adopters programmeNimbus is currently undergoing final testing and will be operational later this year. To assist with the testing, Pawsey is inviting applications for the Nimbus Early Adopter programme. Proposed projects should plan to be completed prior to the start of normal service in August.  Please be aware that allocation size is limited as we prepare to full production,  and the system will be running ‘at risk’ as we complete configuration of Nimbus.   You will be asked to provide a short report at the completion of your work.

To apply to try out Nimbus in this programme, go here https://apply.pawsey.org.au and fill out a Nimbus Early Adopter Application.

Need more help or training?Pawsey will be running a series of Cloud training events in 2017.

The first event is on 13th June, 2017. This training will introduce users to Nimbus, including:

An introduction to setting up virtual machines Migrating your VMs from other cloud services Advanced tools available in Nimbus

To register please visit here.

In addition, a series of help clinics shall be held at each of the Universities over June 2017. Detail. will be posted very soon on our training webpage so please keep a lookout for updates.

For all other Pawsey events, please see the Pawsey Events schedule here

VISUALISATION

A petascale supercomputer enables a researcher to generate and subsequently analyse data of unprecedented size and complexity, but this is only the first step in the research process.

Data visualisation (the process of applying advanced algorithms and computer graphics to data) allows researchers to better understand their raw data and translate it into meaningful forms.

These new insights allow researchers to learn something new, to work faster than using simpler techniques to more rapidly identify problems, and to communicate with peers or with a wider audience using visual displays.

Services AvailableThe Pawsey Supercomputing Centre Visualisation team provides a package of hardware, software and expertise, to tackle the whole range of visualisation activities that are relevant to the Pawsey Supercomputing Centre user community.

Applying visualisation techniques to difficult datasets can require specialist hardware, which can include high end graphics cards for handling large datasets in real-time, novel display technologies to fully exploit the human visual system, and user interface devices to facilitate the interaction. The Pawsey Supercomputing has invested in the following technologies in support of visualisation, they are based across the Pawsey Supercomputing Centre partners but may be freely used by a researcher at any of the partner sites.

Stereoscopic 3D displays Immersive hemispherical displays

High resolution displays High performance graphics workstations Spherical display Specialist still capture cameras Specialist video cameras 3D capture technology User interface and tracking systems Commercial licenses for a high end visualisation software

packages.

How to AccessOur visualisation equipment can be booked and in some cases borrowed via the Pawsey Supercomputing Centre. Please email here to be put in contact with the necessary staff.

Visualisation Locations

Please click the images below for further information on the visualisation resources available at each of the partner locations.

CSIRO VISUALISATIONThe Pawsey CSIRO ARRC partner provides expertise in visualisation, virtual environments, specialised visualisation hardware, stereoscopic display systems and software video conferencing.

Current projects have an emphasis on geoscience disciplines; such as 3D seismic visualisation and volumetric visualisation of microCT rock scan data. However, researchers in other disciplines are encouraged to discuss their requirements with the facility staff.

Display options include monitor and projector based stereoscopic 3D systems. The latter is offered in visualisation lab configuration with 10 seat capacity.

ResourcesThe Pawsey CSIRO ARRC visualisation facilities include various fixed and portable visualisation display and capture equipment.

Christie Mirage 4K25 stereoscopic visualisation system in the Pawsey GWC Interpretive Centre

Room-based visualisation facilities are:

the ARRC NGL Vis Lab, located in the ARRC Building, comprises a high resolution 4K x 4K stereoscopic 3D floor to ceiling wall complete with head tracking and 3D input devices.

the GWC Interpretive Centre , located in the Pawsey Centre building, comprises a high resolution, large screen rear-projected 4K stereoscopic visualisation system.

Standalone workstations available for visualisation processing and analysis use at the ARRC are:

Dell T7600 Precision Windows 7 64-bit workstation that contains dual eight-core 3.10 GHz CPUs, 256GB RAM (192 GB accessible by OS) and 6 GB Nvidia Quadro 6000 graphics card.

Dell T7610 Precision Windows 8.1 64-bit workstation that contains dual 12-core 2.7 GHz CPUs, 256 GB RAM,4 GB Nvidia Quadro K5000 graphics card and 5 GB Tesla K20 GPU Accelerator card.

 

Portable display systems available for loan from the ARRC are:

24″ MagicPlanet sphere-shaped display 46″ Samsung 3D TV with 3D Blu-ray player and 3D glasses. 27″ True3d stereoscopic display monitor system with 3D glasses

Portable data capture items for loan from the ARRC include:

Panasonic AG-3DA1 professional grade 3D camcorder 5DT data gloves (pair of left and right hand) ASL MobileEye tetherless eyetracker system

Software:

The Pawsey Supercomputing Centre currently funds a standalone Avizo Fire license and a bundled Avizo Fire-Avizo Earth license pair, which can be accessed and run in the Vis Lab and on the Vis workstation computers.

A selection of commonly used open source visualisation and analysis software is also loaded on the Vis computers and other commercial software for which researchers provide their own licenses are installed when appropriate and as required.

 

For more information, please contact:

Dr Andrew SquelchSenior Visualisation SpecialistAustralian Resources Research Centre26 Dick Perry AvenueTechnology ParkKensingtonWA 6151Australia

Phone: +61 8 6436 8725Fax: +61 8 6436 8555Email: andrew.squelch@pawsey.org.au

PAWSEY / ERESEARCH AT ECUThe Pawsey ECU partner provides a broad range of visualisation services in support of research within Edith Cowan University and across the other Pawsey Supercomputing Centre partners. Researchers with visualisation problems are encouraged to contact the Centre to discuss their requirements. The ECU facility provides some specific hardware resources along with associated expertise that include, but not limited to, the following.

Video Conferencing 3D Stereo Projection system High end workstations iDome (new in 2013) Motion Capture technology

 

For more information please contact Darren Gibson or Heather Boyd by email (eresearch@ecu.edu.au) or phone (08 6304 6767). The visualisation centre can be found in Room 13.118 on the Mount Lawley Campus.

MURDOCH VISUALISATIONThe Bioinformatics Research Laboratory (BRL) has an immersive display wall of more than 18 megapixels. Display wall solutions are used for high-resolution imagery, animation, information display or visualisation.

Such functions are well suited to the analysis of complex biological data where both high volumes of data and high levels of detail are required simultaneously for effective interaction.

The Display Wall’s primary benefit is to allow groups to collaboratively and simultaneously examine highly detailed imagery and large amounts of information in a coordinated way.

The Display Wall comprises a matrix of data projectors carefully aligned to throw a tiled image onto the rear of a custom screen. A cluster of powerful computers are required to drive the display. Each computer renders graphics data to workstation levels of detail and controls several projectors.

The system requires a fast network interface to existing data management systems that is capable of retrieving and manipulating large amounts of information at once.

For more information, please contact Professor Matt Bellgard by email (mbellgard@ccg.murdoch.edu.au) or phone (08 9360 6088).

The Pawsey Murdoch partner is in the Centre for Comparative Genomics, third floor of the BITL building.

UWA VISUALISATIONVisualisation is the process of providing insight into datasets by the application of advanced computer graphics.

The Pawsey UWA partner provides a broad range of visualisation services in support of research within The University of Western Australia and across the other Pawsey Supercomputing Centre partners. Researchers with visualisation problems are encouraged to contact the Centre to discuss their requirements. The Pawsey UWA partner provides some specific hardware resources along with associated expertise that include, but not limited to, the following:

High resolution 4K x 4K stereoscopic 3D floor to ceiling wall complete with head tracking and 3D input devices.

Immersive projection environment known as the iDome. High resolution tiled display. Various unique image and video capture devices. Experimental and novel user interfaces including 3D camera

trackers,tablets and gesture input devices.

Novel presentation techniques including rapid prototyping and holograms.

High end workstations and licenses to commercial visualisation packages.

For more information, please contact visualisation@pawsey.org.au

The visualisation laboratory can be found on the ground floor of the UWA Physics Building.

TRAININGPawsey Supercomputing Centre offers a series of training courses every year. These courses range from introductory sessions on Pawsey, Supercomputing, Cloud computing and Linux, to more advanced sessions on MPI, OpenMP and GPU programming. Courses on visualisation and data are offered as well. To find out more details on the course descriptions, please visit Pawsey User Portal.

The schedule of training is usually published for the whole year at the beginning of each year, but dates and venues may be subject to change. Please come back here again and check on the status as it gets closer to the dates.

All Pawsey events, including the trainings listed here, are also made available on the Events  page and circulated around through the Pawsey Friends mailing list — subscribe here  to keep updated!

2017 Training Calendar13-15 February Pawsey User Training — Introductory and Intermediate Course13-17 March Pawsey User Training — Introductory, Intermediate and Developer Course8-10 May Pawsey User Training — Introductory and Intermediate Course10-12 July Pawsey User Training — Introductory and Intermediate Course11-15 September Pawsey User Training — Introductory, Intermediate and Developer Course13-15 November Pawsey User Training — Introductory and Intermediate Course

Note: The above calendar is provisional and the dates and venues may be subject to change. Please visit this page again as it gets nearer to the dates.

2016 Training Calendar8-9 February Introductory and Intermediate Supercomputing Training14-16 March Introductory and Intermediate Supercomputing Training11-13 May Introductory and   Intermediate Supercomputing Training 11-13 July Introductory and Intermediate Supercomputing Training14-15 July Developer Supercomputing Training12-14 September Introductory and Intermediate Supercomputing Training7-9 November Introductory and Intermediate Supercomputing Training

Note: The above calendar is provisional and the dates and venues may be subject to change. Please visit this page again as it gets nearer to the dates.

Download Course MaterialsTo download materials for the courses on offer, please visit the Pawsey User Portal.

Who Should Attend? The target Audience is current and future users of

Pawsey’s computational resources, particularly the supercomputers.

Attendees need to have computer literacy. ***To attend Developer courses you should be experienced with

programming in C/C++ or Fortran. The Introduction to Linux session is strongly recommended for

those who are not familiar with Linux.

The training courses that the Pawsey Supercomputing Centre regularly gives to academia are also available to industry and government. We are also able to provide tailored courses at customer premises where there is interest from a larger group of people.

For more information, please email: help@pawsey.org.au

RESEARCHER FEEDBACK 20162nd March 2017

Results from our annual Pawsey Researcher Survey for 2016 are back!

Pawsey is pleased to announce that feedback indicates that “researchers using Pawsey facilities have an overall satisfactory experience of services, staff, training, and events.” This makes the sixth consecutive year in which Pawsey “meets expectations” for almost all of its researchers with Pawsey providing consistent “good service” for our user base.

The survey results suggested that while certain areas could be improved upon, no glaring issues were apparent. Pawsey staff have used the information from the survey to form strategies for enhancement to services, and earmark key areas for upcoming change and transformation.

 

Some specific examples of the positive feedback received include:

“Good, efficient events and communications” “Helpful and hard-working staff” “Generally well integrated services throughout” “Overall a good cloud service”

 

Areas for improvement highlighted:

More advanced training and increased online materials (especially for out of State researchers).

Pawsey is pleased to announce that new online and advanced training offerings will be available during 2017. This will extend the availability of training courses to support learners across Australia. The three existing main training units (introductory, intermediate and advanced) have been divided into 10-20 slide modules, each with a stand-alone topic, e.g. how to transfer files or check group usage. The modules will then be converted into 10-30 minute videos and will be made available on Pawsey websites and publicly on YouTube (to be confirmed). Each video will also have both lecture-style talk and hands-on demonstrations.

With many modules being planned for release, Pawsey will initially give priority to Pawsey-specific introductory and intermediate modules which cover basic supercomputing, data and visualisation topics. The next batch of modules will cover more advanced topics such as serial code optimisation, parallelising with MPI, and parallelising with OpenMP. Further details will be provided as soon as release dates are confirmed.

 

Increased machine learning capabilities and cloud storage

At the start of July 2017, Pawsey will deploy a new OpenStack cluster to best meet the needs of researchers. Specifically, the new service will support large data workflows, and computational tasks. The new resource will support / provide:

1. HPC on OpenStack – For any compute job that doesn’t fill nodes on Magnus.

2. Data Analytics using Sahara – Sahara is an OpenStack Module for easy deployment and management of DA clusters and jobs. This will remove the need to deploy complex Data Analytics clusters.

3. Machine Learning to automatically investigate complex relationships within a dataset.

4. Data Centric Workflows – Researchers with large, inhomogeneous datasets require adaptable compute facilities for data processing workflows.

As soon as more information is available, the new cloud service will be promoted to ensure all researchers can take full advantage of this new resource.

 

Increased tape storage, replace tape with spinning disk; more storage generally

There is presently a licence to store up to 100PB of data in the Pawsey Centre on tape and independently up to 6PB of data on high performance enterprise grade spinning disk.  There is presently capacity to meet the immediate needs and forecast long term requirements of researcher projects.  Researchers with data needs are invited to apply for storage at any time using the data portal.

 

Improve data transfer capability within data stores, including interactions with Mediaflux data portal and the command line tools

 In December, Pawsey staff evaluated and improved some ethernet configurations, which have decreased various data transfer times.   In 2017, the team will be investigating leverage Mediaflux’s native DMF support to seamlessly improve the efficiency of command line data transfers involving the HSM.

 

Improve integration and ability to move data around

Pawsey current offers two dedicated systems to long-term storage provided via a POSIX filesystems, supporting copy tools to move data in and out. During 2017, staff will investigate running a SLURM daemon on both systems to allow researchers to schedule jobs on the system. Radio astronomer researchers are presently beta testing a new dcp, a high performance copy tool for Lustre; this will hopefully become widely available in quarter one of 2017. Staff plan to investigate the feasibility of using Mediaflux’s S3 interface (via Ceph) to provide collection access to OpenStack Virtual Machines for enhanced data processing services.

 

Improved HPC reliability (reducing the number of interrupts to workflow)

Pawsey staff are at present reviewing the number of maintenance days held annually, as well as investigating the possibility of running more maintenance while the systems are operational. Once a decision is made an updated schedule will be available here

The Cray Sonexion serving /scratch and /scratch2 have been upgraded to the latest stable release (NEO 2.0). SLURM has been upgraded to 16.05.8 and will be the major version that Magnus and Galaxy will use until their retirement. Cray Linux Environment has been stable at 5.2UP04 since last year and will only be upgraded to rectify known issues.

 

Faster scratch disk

/scratch also uses NEO 2.0, but the underlying Lustre has been upgraded from version 2.1.x to 2.5.x which sees a number of software improvements such as improved meta-data performance and single threaded performance. A new /group filesystem will be online in March 2017 which will double the amount of storage space and triple the performance of the current filesystem. This is also based around a much newer version of Lustre (2.7.x) which has a number of performance improvements which will be most clearly seen on the Zeus data mover nodes.

 

Deletion warnings needed

Deletion warnings will be investigated during 2017.

 

Increased GPU capability

The Advance Technology Cluster will be available in quarter two of 2017. Researchers will have access to 44 NVIDIA Pascal GPUs which a much more modern software stack based around SLES 12 and CUDA 8+. This will allow for the support of projects requiring Machine Learning or Deep Learning software. Which the Zeus Cluster expansion coming online in Q3 2017, the current nodes will be allocated to a dedicated to either a GPU queue or visualisation queue depending on whether the GPU is Tesla or Quadro based. This will allow Pawsey to resume having a dedicated GPU service and be able to allocate resources to them (rather than the current shared-service offering).

 

HPC resource allocations more than once a year

Increasing the resource allocations was tried and tested during 2015-2016. Unfortunately it was largely unsuccessful. In addition, feedback from the Chairs of the location allocation committees indicated that they preferred to have one calendar year intake. However Pawsey will contact the committees again and indicate that researchers have raised this issue again.

 

New services for Supercomputing

In 2017, priority new services such as ability to add users to projects and the evaluation of Shifter (a Docker-like technology) on all Supercomputing systems will be investigated.

 

Pawsey extends a warm thank you to all the survey participants. We look forward to the further advancement of our services and facilities made possible through valuable feedback offered from researchers. We encourage all researchers to continue providing suggestions via surveys so that Pawsey may best serve the research needs of Australia.

Researchers are invited to provide further feedback at any time by contacting feedback@pawsey.org.au