- 245
- 905 727
Sharcnet HPC
Canada
Registrace 13. 09. 2012
This channel provides public access to recordings of SHARCNET training events. These recordings are part of a presentation series organized by SHARCNET staff. For more information about SHARCNET events visit our calendar at www.sharcnet.ca/my/news/calendar.
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers. With this infrastructure we enable world-class academic research. We aim to:
accelerate computational academic research,
attract the best students and faculty to our partner institutions by providing cutting edge expertise and hardware,
and link academic researchers with corporate partners in a search for new business opportunities
SHARCNET is a partner organization of Compute Ontario and Digital Research Alliance of Canada national advanced research computing platform.
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers. With this infrastructure we enable world-class academic research. We aim to:
accelerate computational academic research,
attract the best students and faculty to our partner institutions by providing cutting edge expertise and hardware,
and link academic researchers with corporate partners in a search for new business opportunities
SHARCNET is a partner organization of Compute Ontario and Digital Research Alliance of Canada national advanced research computing platform.
Using machine learning to predict rare events
In some binary classification problems, the underlying distribution of positive and negative samples are highly unbalanced. For example, fraudulent credit card transactions are rare compared to the volume of legitimate transactions. Training a classification model in such a case needs to take into account the nature of skewed distribution. In this seminar, we will develop a fraud detector which can be used to screen credit card transactions. We will describe the methods used to handle unbalanced data training.
_________________________________________________
This webinar was presented by Weiguang Guan (SHARCNET) on August 28th, 2024, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SHARCNET. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: www.computeontario.ca/training-colloquia . Recordings, slides, and other materials can be found here: helpwiki.sharcnet.ca/wiki/Online_Seminars
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (www.sharcnet.ca). SHARCNET is a part of Compute Ontario (computeontario.ca/) and Digital Research Alliance of Canada (alliancecan.ca).
_________________________________________________
This webinar was presented by Weiguang Guan (SHARCNET) on August 28th, 2024, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SHARCNET. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: www.computeontario.ca/training-colloquia . Recordings, slides, and other materials can be found here: helpwiki.sharcnet.ca/wiki/Online_Seminars
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (www.sharcnet.ca). SHARCNET is a part of Compute Ontario (computeontario.ca/) and Digital Research Alliance of Canada (alliancecan.ca).
zhlédnutí: 61
Video
Diagnosing Wasted Resources from User Facing Portals on the National Clusters
zhlédnutí 33Před 21 hodinou
Researchers often leave resources on the table when specifying their job requirements on the national systems. This talk builds on previous sessions and uses the Digital Research Alliance of Canada's User Facing Portals to explore what different types of jobs look like when they waste resources. Demonstrations will include interactive jobs, parallel jobs, GPU workflows, and more. With more accu...
The Emergence of WebAssembly (Wasm) in Scientific Computing
zhlédnutí 134Před dnem
Developed collaboratively by major browser vendors, including Mozilla, Google, Microsoft, and Apple, WebAssembly (Wasm) addresses the limitations of traditional web programming languages like JavaScript. But what makes it so compelling for scientists? First, Wasm allows code written in languages like C/C , Fortran or Rust to be compiled into its instruction format and run directly in the browse...
Exploring Compute Usage from User Facing Portals on the National Clusters
zhlédnutí 96Před měsícem
Previous seminars in this series have described using Python tools to explore job properties and usage characteristics on the Digital Research Alliance of Canada general purpose compute clusters. The end goal of exploring job properties and usage characteristics is to get the most out of the resources available to research accounts and to minimize wait times in the job queue. This seminar revie...
Compute Ontario Summer School 2024
zhlédnutí 594Před 3 měsíci
UPDATE: registration is now open for Compute Ontario Summer School (June 3-21, 2024): training.computeontario.ca/coss2024.php In this colloquium, we will present the curriculum of the 2024 Compute Ontario Summer School, to be held from the 3rd to the 21st of June. Jointly organized by the Centre for Advanced Computing, SciNet, SHARCNET, and in collaboration with the Research Data Management Net...
Data Wrangling with Tidyverse (part 2)
zhlédnutí 95Před 4 měsíci
Tidyverse is an cohesive set of packages for doing data science in R. In an earlier talk, we began reviewing the data munging portions of tidyvese (dplyr, forcats, tibble, readr, stringr, tidyr, and purr) by using it to reconstruct the data hierarchy in a 500 pages reference PDF given only the words on each page and their bounding boxes. This talk will complete this. If you have not seen the fi...
Accelerating data analytics with RAPIDS cuDF
zhlédnutí 149Před 4 měsíci
Pandas, renowned as the go-to library for data manipulation and analysis in Python and widely adopted in machine learning. However, Pandas is slow. With the introduction of NVIDIA cuDF.pandas, the accelerated power of GPUs is integrated into Pandas, enabling faster processing without the need for any code changes. A live demo will showcase this enhancement on clusters. This webinar was presen...
Accelerating Graph Analysis on GPUs
zhlédnutí 357Před 5 měsíci
Graph analysis plays a critical role in many applications across various domains, ranging from social network analysis to bioinformatics, to fraud detection, to cybersecurity, to recommendation systems, etc. NetworkX is the go-to library for graph analysis in Python. However, when dataset and graph sizes grow, the performance of using NetworkX becomes a significant concern. This webinar introdu...
Make: a declarative, lazy, parallel workload manager. Elegant or obsolete?
zhlédnutí 86Před 5 měsíci
Make is a classic Unix development tool, which may seem archaic and narrow-purpose. But if you think of it as a declarative, parallelized workflow automation tool, it sounds more relevant. We'll consider stereotypical use of make, then its general properties, and show some interesting examples of make applied to unusual uses. This webinar was presented by Mark Hahn (SHARCNET) on March 13th, 2...
Debugging your code with DDT
zhlédnutí 173Před 6 měsíci
One of the important steps of developing or maintaining a code is debugging: checking the code for errors. Simple toy codes can be debugged using print statements, but realistic codes need specialized debugging tools. We have a powerful debugger "DDT" installed on Graham and Niagara clusters. This presentation will walk you through the steps required to start debugging your codes using DDT, and...
MySQL Part 3: Constraints and Joins
zhlédnutí 96Před 6 měsíci
In MySQL, constraints and joins are fundamental concepts used to ensure data integrity in a database and query data from multiple tables. Constraints are rules enforced on the data columns of a table. Constraints provide the accuracy and reliability of the data within a database. Joins in MySQL combine rows from two or more tables based on a related column. Previous parts in the series: * Part ...
Introduction to GPU programming with OpenMP
zhlédnutí 382Před 7 měsíci
OpenMP is a popular, portable and widely supported shared-memory parallel programming model in HPC. The OpenMP API supports multi-platform parallel programming in C/C and Fortran. As computer hardware has grown to include GPU and other specialized accelerators, OpenMP has grown as well to add device support for parallel programming on GPU and accelerators. This seminar will give an introduction...
False Sharing and Contention in Parallel Codes
zhlédnutí 136Před 7 měsíci
Sequential programs can repeatedly read from and write to memory locations seemingly without issues. On the other hand, parallel programs can easily fall prey to weird behaviours resulting in small to very significant issues and/or performance loss that are not always easily attributable to specific pieces of code one has written. Such behaviours can be seen in multithreaded C, C , Fortran, Ope...
Skorch: Training PyTorch models with scikit-learn
zhlédnutí 476Před 9 měsíci
PyTorch is an enormously popular framework for developing deep learning models in Python. However, scikit-learn is one of the most popular libraries for general machine learning. Skorch is a wrapper for PyTorch that allows one to use models written with PyTorch with the scikit-learn library. In this talk, we will explore how skorch allows for PyTorch models to be easily incorporated into scikit...
Squeeze more juice out of a single GPU in deep learning
zhlédnutí 211Před 9 měsíci
It’s well known that GPUs can significantly accelerate neural network training. However, not everyone knows that a single GPU is sufficient to train most neural networks except for a few large ones (like LLM). In fact, a GPU is under-utilized in most cases. In this talk, we are addressing the under-utilization issue and proposing a way to make full use of the GPU capacity. The goal is to increa...
Generalized End to End Python and Neuroscience Workflows on a Compute Cluster
zhlédnutí 122Před 10 měsíci
Generalized End to End Python and Neuroscience Workflows on a Compute Cluster
p2rng - A C++ Parallel Random Number Generator Library for the Masses
zhlédnutí 128Před 10 měsíci
p2rng - A C Parallel Random Number Generator Library for the Masses
Exploring job wait times on Alliance compute clusters: a holistic view
zhlédnutí 96Před 11 měsíci
Exploring job wait times on Alliance compute clusters: a holistic view
Automating scientific workflows with AiiDA
zhlédnutí 292Před rokem
Automating scientific workflows with AiiDA
DIY job monitoring, from cache misses to CO2 footprint
zhlédnutí 105Před rokem
DIY job monitoring, from cache misses to CO2 footprint
Leveraging the power of Linux on Windows with WSL
zhlédnutí 304Před rokem
Leveraging the power of Linux on Windows with WSL
Modern Approaches to Profiling in Python with Scalene
zhlédnutí 3,9KPřed rokem
Modern Approaches to Profiling in Python with Scalene
CUDA, ROCm, oneAPI - All for One or One for All?
zhlédnutí 3,3KPřed rokem
CUDA, ROCm, oneAPI - All for One or One for All?
Before and after submitting Octave/Matlab jobs on the clusters
zhlédnutí 151Před rokem
Before and after submitting Octave/Matlab jobs on the clusters
Plotnine: R's Grammar of Graphics in Python
zhlédnutí 761Před rokem
Plotnine: R's Grammar of Graphics in Python
Accelerated DataFrame with Dask-cuDF on multiple GPUs
zhlédnutí 903Před rokem
Accelerated DataFrame with Dask-cuDF on multiple GPUs
An introduction to MPLAPACK, a multi-precision linear algebra library
zhlédnutí 362Před rokem
An introduction to MPLAPACK, a multi-precision linear algebra library
Thank you so much! I know this is pretty straightforward stuff, but 54:33 really helped me out fix an issue with containerized experiments.
DBInterface is not an interface... I was wondering how you might create an interface in javascript since it's not natively supported. Typescript would be a better choice for this if you really wanted this, but as this is, this "interface" contains implementation, something an interface is not, by definition, supposed to do.
Where's the github you mention would be available?
how can i install these GPU Utilization packages(GPU dashboard)?
Obrigada pelo vídeo incrível! 🇧🇷
Nice explanation. I converted my 2D parallelized fortran code to 3D and now it gives me NaN, any suggestions please?
thanks. One of the better videos on collaborative groups.👌
Be like hay.
Oh, my goodness.
😮 It must not be the right way to do it, because cythin should be at least 40 times faster. He should probably get out of the juoyter notebook and get better, more reasonable measurements.
Is there cuda fortran for windows?
what do you mean? The compiler? yes
@@muhammadfaridkhandaq4379 thanks
This was nice. Thank you.
hi guys, in 57:11, after "ssh -A -J narval nc10201", do you know why I still got required the password, and then got this error: "Received disconnect from UNKNOWN port 65535:2: Too many authentication failures Disconnected from UNKNOWN port 65535" ? Thank you in advance
Is it possible to automatically get compute node id and connect to it?
Not necessarily new … we did this with the mainframe and the Cray Vector processors … GPUs are great at vector math and solving simultaneous equations
Excellent talk - great content, very informative, very useful! Thank you.
Fantastic presentation, thank you so much! What about xperf on windows environment? Still relevant?
Amazing job!
Thanks for sharing useful contents. It would have been better if you had named it "faculty member edition", cause it is specifically for supervisors.
This deserves 10K thumbs up, not just 39!!!
Nice presentation and useful instructions. Thanks a lot!
Your cython explanation and tutorial is the most crystal clear I've heard...and I have listened to > 10 videos! Excellent content! Thank you for this, it makes things less painful ❤
In the code mentioned at 22:50, the for loop initialisation condition must be g.size()/2 right?. To accomodate any thread group size.
Thank you for the presentation 😊
thanks for the video. Its really helpful I had a doubt, so say you want to train an xgboost model on top of a 200GB dataset and I have a VM with some gpus having a total combined memory of ~100GB (GPU memory + VM memory). Will I be able to train the model successfully on that using LocalCUDACluster .
did you try?
Great tutorial. May I ask you a question? If I have to use an old version of Node for my project that does not support async/await but supports promises, how can I rewrite the "for" loop on 19:16? I need to sustain the ordering of the promises, which means that just start them in loop with "then()" attached to every one of them is not a solution. Also want to ask you - in one of my old web apps I used an inline <script> block in into which I uploaded a file or some web form for new item creation, that script was something like "if (window.parent) { window.parent.doStuff(data) }". It worked fine even in IE6+, but is it a clean approach?
Great video, helped connect to UMIACS HPC
Thanks a lot. It is a very well structured nice video. Without examples it is very hard to learn programming. Thanks a lot. 😄
It was really a nice demonstration for CUDA. Thanks
Glad you liked it!
Very useful !! specially the explanation of the color coding
Glad it was helpful!
is the source code available for the debugger part?
Any idea of when H100 will be available on Graham?
We plan to replace graham with a new cluster in 2024 (if funding comes through), and this new cluster may contain H100 GPUs. Before then we may get a small number of H100 GPUs for testing purposes.
@@pawelpomorski4025 I'm interested in testing H100 GPUs from a researcher's perspective when they are available. 😁
@@dakezhang2845 There is a lot of interest. When we get some H100 cards for testing, we will advertise it, probably in our monthly newsletter which all users receive.
@@pawelpomorski4025 What happens to the old cluster? Can research groups purchase the old cores or GPUs?
@@ameerracle The cluster hardware is owned by the university, which decides how to dispose of it if we (SHARCNET) retire it. Anyway, it looks like graham will be running with the current hardware at least until early 2025.
how to profile in production using scalene pls
i need the steps to take to get this done
Do you think the AI programmer community will support/adopt RocM and to an extent AMD DataCenter GPUs?
RocM uses HIP which is very similar to CUDA, so any program written in CUDA can be ported to AMD (assuming it does not use any advanced features which are exclusive to NVIDIA GPUs). As for data centres, given the high demand and expense of NVIDIA GPUs, more AMD data center GPUs may be adopted.
Can I use this code?
Thank you for the great explanation 🙏
My pleasure!
Well done. I hope you'll make a follow up about a larger python program, with classes, various files, etc. The challenge I find is that a profiler is not so helpful once the work spreads out across components. I need something more specific to say "how can this function call be more efficient"?
24:38 How are you configuring Jupyterhub like this in the background connecting to these cluster resources. Is there documentation on this available or open source github that I can take a look at?
awesome. thanks man
Glad it helped!
'/^seq2$;/
awk '{print $1, $3, $5} filename
Thanks a lot for sharing that. It helped me connect UGent's HPC
Glad it helped
How to overcome with this issue:-RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1670525552843/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, i nternal error, NCCL version 2.14.3 ncclInternalError: Internal check failed. Last error: Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 1a000
same error bro, did u rectified it?
Thanks for the presentation. Please make MIG available for everyone
If you are interested in trying our experimental MIG setup on narval, please submit a ticket with a request. We might be able to give you some access in January.
Dear Ge, thanks for the video. Dou you know if its possible to debug in many nodes ?
The first example shown will not use multiple GPUs, only one. You can see at 27:36 that in the code (it would have been nice to have line numbers) [device = 'cuda:0'...] meaning it only uses the first GPU. I believe this should be [device ='cuda:0,1'...]
u are correct
Can you tell exactly how to use or provide syntax to utilize multi gpu
Thank you for the tutorial and very educational.
You're very welcome!
Is there an openGL version of gnuPLOT? So others wouldn't need to install gnuPlot to din my program?
Thanks a lot.
Most welcome!
great talk ! Can we use Nsights for cluster wide profiling ?