Sharcnet HPC
Sharcnet HPC
  • 245
  • 905 727
Using machine learning to predict rare events
In some binary classification problems, the underlying distribution of positive and negative samples are highly unbalanced. For example, fraudulent credit card transactions are rare compared to the volume of legitimate transactions. Training a classification model in such a case needs to take into account the nature of skewed distribution. In this seminar, we will develop a fraud detector which can be used to screen credit card transactions. We will describe the methods used to handle unbalanced data training.
________________________________________­_________
This webinar was presented by Weiguang Guan (SHARCNET) on August 28th, 2024, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SHARCNET. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: www.computeontario.ca/training-colloquia . Recordings, slides, and other materials can be found here: helpwiki.sharcnet.ca/wiki/Online_Seminars
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (www.sharcnet.ca). SHARCNET is a part of Compute Ontario (computeontario.ca/) and Digital Research Alliance of Canada (alliancecan.ca).
zhlédnutí: 61

Video

Diagnosing Wasted Resources from User Facing Portals on the National Clusters
zhlédnutí 33Před 21 hodinou
Researchers often leave resources on the table when specifying their job requirements on the national systems. This talk builds on previous sessions and uses the Digital Research Alliance of Canada's User Facing Portals to explore what different types of jobs look like when they waste resources. Demonstrations will include interactive jobs, parallel jobs, GPU workflows, and more. With more accu...
The Emergence of WebAssembly (Wasm) in Scientific Computing
zhlédnutí 134Před dnem
Developed collaboratively by major browser vendors, including Mozilla, Google, Microsoft, and Apple, WebAssembly (Wasm) addresses the limitations of traditional web programming languages like JavaScript. But what makes it so compelling for scientists? First, Wasm allows code written in languages like C/C , Fortran or Rust to be compiled into its instruction format and run directly in the browse...
Exploring Compute Usage from User Facing Portals on the National Clusters
zhlédnutí 96Před měsícem
Previous seminars in this series have described using Python tools to explore job properties and usage characteristics on the Digital Research Alliance of Canada general purpose compute clusters. The end goal of exploring job properties and usage characteristics is to get the most out of the resources available to research accounts and to minimize wait times in the job queue. This seminar revie...
Compute Ontario Summer School 2024
zhlédnutí 594Před 3 měsíci
UPDATE: registration is now open for Compute Ontario Summer School (June 3-21, 2024): training.computeontario.ca/coss2024.php In this colloquium, we will present the curriculum of the 2024 Compute Ontario Summer School, to be held from the 3rd to the 21st of June. Jointly organized by the Centre for Advanced Computing, SciNet, SHARCNET, and in collaboration with the Research Data Management Net...
Data Wrangling with Tidyverse (part 2)
zhlédnutí 95Před 4 měsíci
Tidyverse is an cohesive set of packages for doing data science in R. In an earlier talk, we began reviewing the data munging portions of tidyvese (dplyr, forcats, tibble, readr, stringr, tidyr, and purr) by using it to reconstruct the data hierarchy in a 500 pages reference PDF given only the words on each page and their bounding boxes. This talk will complete this. If you have not seen the fi...
Accelerating data analytics with RAPIDS cuDF
zhlédnutí 149Před 4 měsíci
Pandas, renowned as the go-to library for data manipulation and analysis in Python and widely adopted in machine learning. However, Pandas is slow. With the introduction of NVIDIA cuDF.pandas, the accelerated power of GPUs is integrated into Pandas, enabling faster processing without the need for any code changes. A live demo will showcase this enhancement on clusters. ­ This webinar was presen...
Accelerating Graph Analysis on GPUs
zhlédnutí 357Před 5 měsíci
Graph analysis plays a critical role in many applications across various domains, ranging from social network analysis to bioinformatics, to fraud detection, to cybersecurity, to recommendation systems, etc. NetworkX is the go-to library for graph analysis in Python. However, when dataset and graph sizes grow, the performance of using NetworkX becomes a significant concern. This webinar introdu...
Make: a declarative, lazy, parallel workload manager. Elegant or obsolete?
zhlédnutí 86Před 5 měsíci
Make is a classic Unix development tool, which may seem archaic and narrow-purpose. But if you think of it as a declarative, parallelized workflow automation tool, it sounds more relevant. We'll consider stereotypical use of make, then its general properties, and show some interesting examples of make applied to unusual uses. ­ This webinar was presented by Mark Hahn (SHARCNET) on March 13th, 2...
Debugging your code with DDT
zhlédnutí 173Před 6 měsíci
One of the important steps of developing or maintaining a code is debugging: checking the code for errors. Simple toy codes can be debugged using print statements, but realistic codes need specialized debugging tools. We have a powerful debugger "DDT" installed on Graham and Niagara clusters. This presentation will walk you through the steps required to start debugging your codes using DDT, and...
MySQL Part 3: Constraints and Joins
zhlédnutí 96Před 6 měsíci
In MySQL, constraints and joins are fundamental concepts used to ensure data integrity in a database and query data from multiple tables. Constraints are rules enforced on the data columns of a table. Constraints provide the accuracy and reliability of the data within a database. Joins in MySQL combine rows from two or more tables based on a related column. Previous parts in the series: * Part ...
Introduction to GPU programming with OpenMP
zhlédnutí 382Před 7 měsíci
OpenMP is a popular, portable and widely supported shared-memory parallel programming model in HPC. The OpenMP API supports multi-platform parallel programming in C/C and Fortran. As computer hardware has grown to include GPU and other specialized accelerators, OpenMP has grown as well to add device support for parallel programming on GPU and accelerators. This seminar will give an introduction...
False Sharing and Contention in Parallel Codes
zhlédnutí 136Před 7 měsíci
Sequential programs can repeatedly read from and write to memory locations seemingly without issues. On the other hand, parallel programs can easily fall prey to weird behaviours resulting in small to very significant issues and/or performance loss that are not always easily attributable to specific pieces of code one has written. Such behaviours can be seen in multithreaded C, C , Fortran, Ope...
Skorch: Training PyTorch models with scikit-learn
zhlédnutí 476Před 9 měsíci
PyTorch is an enormously popular framework for developing deep learning models in Python. However, scikit-learn is one of the most popular libraries for general machine learning. Skorch is a wrapper for PyTorch that allows one to use models written with PyTorch with the scikit-learn library. In this talk, we will explore how skorch allows for PyTorch models to be easily incorporated into scikit...
Squeeze more juice out of a single GPU in deep learning
zhlédnutí 211Před 9 měsíci
It’s well known that GPUs can significantly accelerate neural network training. However, not everyone knows that a single GPU is sufficient to train most neural networks except for a few large ones (like LLM). In fact, a GPU is under-utilized in most cases. In this talk, we are addressing the under-utilization issue and proposing a way to make full use of the GPU capacity. The goal is to increa...
Generalized End to End Python and Neuroscience Workflows on a Compute Cluster
zhlédnutí 122Před 10 měsíci
Generalized End to End Python and Neuroscience Workflows on a Compute Cluster
p2rng - A C++ Parallel Random Number Generator Library for the Masses
zhlédnutí 128Před 10 měsíci
p2rng - A C Parallel Random Number Generator Library for the Masses
Exploring job wait times on Alliance compute clusters: a holistic view
zhlédnutí 96Před 11 měsíci
Exploring job wait times on Alliance compute clusters: a holistic view
Data Wrangling with Tidyverse
zhlédnutí 201Před 11 měsíci
Data Wrangling with Tidyverse
Automating scientific workflows with AiiDA
zhlédnutí 292Před rokem
Automating scientific workflows with AiiDA
DIY job monitoring, from cache misses to CO2 footprint
zhlédnutí 105Před rokem
DIY job monitoring, from cache misses to CO2 footprint
Leveraging the power of Linux on Windows with WSL
zhlédnutí 304Před rokem
Leveraging the power of Linux on Windows with WSL
Contrastive learning
zhlédnutí 841Před rokem
Contrastive learning
Modern Approaches to Profiling in Python with Scalene
zhlédnutí 3,9KPřed rokem
Modern Approaches to Profiling in Python with Scalene
CUDA, ROCm, oneAPI - All for One or One for All?
zhlédnutí 3,3KPřed rokem
CUDA, ROCm, oneAPI - All for One or One for All?
Running MATLAB on Alliance's Clusters
zhlédnutí 525Před rokem
Running MATLAB on Alliance's Clusters
Before and after submitting Octave/Matlab jobs on the clusters
zhlédnutí 151Před rokem
Before and after submitting Octave/Matlab jobs on the clusters
Plotnine: R's Grammar of Graphics in Python
zhlédnutí 761Před rokem
Plotnine: R's Grammar of Graphics in Python
Accelerated DataFrame with Dask-cuDF on multiple GPUs
zhlédnutí 903Před rokem
Accelerated DataFrame with Dask-cuDF on multiple GPUs
An introduction to MPLAPACK, a multi-precision linear algebra library
zhlédnutí 362Před rokem
An introduction to MPLAPACK, a multi-precision linear algebra library

Komentáře

  • @zappandy
    @zappandy Před 24 dny

    Thank you so much! I know this is pretty straightforward stuff, but 54:33 really helped me out fix an issue with containerized experiments.

  • @karlkrasnowsky1393
    @karlkrasnowsky1393 Před 27 dny

    DBInterface is not an interface... I was wondering how you might create an interface in javascript since it's not natively supported. Typescript would be a better choice for this if you really wanted this, but as this is, this "interface" contains implementation, something an interface is not, by definition, supposed to do.

  • @karlkrasnowsky1393
    @karlkrasnowsky1393 Před 27 dny

    Where's the github you mention would be available?

  • @yusufkemahl5943
    @yusufkemahl5943 Před měsícem

    how can i install these GPU Utilization packages(GPU dashboard)?

  • @FerPerez-mc3wr
    @FerPerez-mc3wr Před měsícem

    Obrigada pelo vídeo incrível! 🇧🇷

  • @idreeskhan-zp5ey
    @idreeskhan-zp5ey Před 2 měsíci

    Nice explanation. I converted my 2D parallelized fortran code to 3D and now it gives me NaN, any suggestions please?

  • @jurgendevlieghere9081
    @jurgendevlieghere9081 Před 2 měsíci

    thanks. One of the better videos on collaborative groups.👌

  • @richardbennett4365
    @richardbennett4365 Před 3 měsíci

    Be like hay.

  • @richardbennett4365
    @richardbennett4365 Před 3 měsíci

    Oh, my goodness.

  • @richardbennett4365
    @richardbennett4365 Před 3 měsíci

    😮 It must not be the right way to do it, because cythin should be at least 40 times faster. He should probably get out of the juoyter notebook and get better, more reasonable measurements.

  • @enriqueaguilarolivares9560
    @enriqueaguilarolivares9560 Před 5 měsíci

    Is there cuda fortran for windows?

  • @Sandeep_Sulakhe
    @Sandeep_Sulakhe Před 5 měsíci

    This was nice. Thank you.

  • @tihihoang
    @tihihoang Před 5 měsíci

    hi guys, in 57:11, after "ssh -A -J narval nc10201", do you know why I still got required the password, and then got this error: "Received disconnect from UNKNOWN port 65535:2: Too many authentication failures Disconnected from UNKNOWN port 65535" ? Thank you in advance

  • @mindlessgreen
    @mindlessgreen Před 6 měsíci

    Is it possible to automatically get compute node id and connect to it?

  • @captaindunsell8568
    @captaindunsell8568 Před 9 měsíci

    Not necessarily new … we did this with the mainframe and the Cray Vector processors … GPUs are great at vector math and solving simultaneous equations

  • @Leonid.Shamis
    @Leonid.Shamis Před 9 měsíci

    Excellent talk - great content, very informative, very useful! Thank you.

  • @camaycama7479
    @camaycama7479 Před 10 měsíci

    Fantastic presentation, thank you so much! What about xperf on windows environment? Still relevant?

  • @AdrianGonzalez-ii7jb
    @AdrianGonzalez-ii7jb Před 11 měsíci

    Amazing job!

  • @akbarravan5604
    @akbarravan5604 Před 11 měsíci

    Thanks for sharing useful contents. It would have been better if you had named it "faculty member edition", cause it is specifically for supervisors.

  • @zgrunschlag
    @zgrunschlag Před 11 měsíci

    This deserves 10K thumbs up, not just 39!!!

  • @Hut-il6oz
    @Hut-il6oz Před rokem

    Nice presentation and useful instructions. Thanks a lot!

  • @pietraderdetective8953

    Your cython explanation and tutorial is the most crystal clear I've heard...and I have listened to > 10 videos! Excellent content! Thank you for this, it makes things less painful ❤

  • @karthikmr416
    @karthikmr416 Před rokem

    In the code mentioned at 22:50, the for loop initialisation condition must be g.size()/2 right?. To accomodate any thread group size.

  • @Eduardo-Quantum
    @Eduardo-Quantum Před rokem

    Thank you for the presentation 😊

  • @AmanSingh-fl3wp
    @AmanSingh-fl3wp Před rokem

    thanks for the video. Its really helpful I had a doubt, so say you want to train an xgboost model on top of a 200GB dataset and I have a VM with some gpus having a total combined memory of ~100GB (GPU memory + VM memory). Will I be able to train the model successfully on that using LocalCUDACluster .

  • @popov654
    @popov654 Před rokem

    Great tutorial. May I ask you a question? If I have to use an old version of Node for my project that does not support async/await but supports promises, how can I rewrite the "for" loop on 19:16? I need to sustain the ordering of the promises, which means that just start them in loop with "then()" attached to every one of them is not a solution. Also want to ask you - in one of my old web apps I used an inline <script> block in into which I uploaded a file or some web form for new item creation, that script was something like "if (window.parent) { window.parent.doStuff(data) }". It worked fine even in IE6+, but is it a clean approach?

  • @user-mo6nn1bw1d
    @user-mo6nn1bw1d Před rokem

    Great video, helped connect to UMIACS HPC

  • @dhimanbhowmick9558
    @dhimanbhowmick9558 Před rokem

    Thanks a lot. It is a very well structured nice video. Without examples it is very hard to learn programming. Thanks a lot. 😄

  • @farzaddizaji7002
    @farzaddizaji7002 Před rokem

    It was really a nice demonstration for CUDA. Thanks

  • @sebastiangomez-wu9gh

    Very useful !! specially the explanation of the color coding

  • @user-gr3fy5bd4c
    @user-gr3fy5bd4c Před rokem

    is the source code available for the debugger part?

  • @dakezhang2845
    @dakezhang2845 Před rokem

    Any idea of when H100 will be available on Graham?

    • @pawelpomorski4025
      @pawelpomorski4025 Před rokem

      We plan to replace graham with a new cluster in 2024 (if funding comes through), and this new cluster may contain H100 GPUs. Before then we may get a small number of H100 GPUs for testing purposes.

    • @dakezhang2845
      @dakezhang2845 Před rokem

      @@pawelpomorski4025 I'm interested in testing H100 GPUs from a researcher's perspective when they are available. 😁

    • @pawelpomorski4025
      @pawelpomorski4025 Před rokem

      @@dakezhang2845 There is a lot of interest. When we get some H100 cards for testing, we will advertise it, probably in our monthly newsletter which all users receive.

    • @ameerracle
      @ameerracle Před 8 měsíci

      @@pawelpomorski4025 What happens to the old cluster? Can research groups purchase the old cores or GPUs?

    • @pawelpomorski4025
      @pawelpomorski4025 Před 8 měsíci

      @@ameerracle The cluster hardware is owned by the university, which decides how to dispose of it if we (SHARCNET) retire it. Anyway, it looks like graham will be running with the current hardware at least until early 2025.

  • @favourkachi2178
    @favourkachi2178 Před rokem

    how to profile in production using scalene pls

  • @ranchuhead4547
    @ranchuhead4547 Před rokem

    Do you think the AI programmer community will support/adopt RocM and to an extent AMD DataCenter GPUs?

    • @pawelpomorski4025
      @pawelpomorski4025 Před rokem

      RocM uses HIP which is very similar to CUDA, so any program written in CUDA can be ported to AMD (assuming it does not use any advanced features which are exclusive to NVIDIA GPUs). As for data centres, given the high demand and expense of NVIDIA GPUs, more AMD data center GPUs may be adopted.

  • @rupjyoti2304
    @rupjyoti2304 Před rokem

    Can I use this code?

  • @fatemehn.nokabadi6232

    Thank you for the great explanation 🙏

  • @pauljohn32
    @pauljohn32 Před rokem

    Well done. I hope you'll make a follow up about a larger python program, with classes, various files, etc. The challenge I find is that a profiler is not so helpful once the work spreads out across components. I need something more specific to say "how can this function call be more efficient"?

  • @sigururbaldursson8118

    24:38 How are you configuring Jupyterhub like this in the background connecting to these cluster resources. Is there documentation on this available or open source github that I can take a look at?

  • @krgrief
    @krgrief Před rokem

    awesome. thanks man

  • @paulosergioschlogl9550

    '/^seq2$;/

  • @paulosergioschlogl9550

    awk '{print $1, $3, $5} filename

  • @abdelra7man87
    @abdelra7man87 Před rokem

    Thanks a lot for sharing that. It helped me connect UGent's HPC

  • @GarimaKumari-sl2gq
    @GarimaKumari-sl2gq Před rokem

    How to overcome with this issue:-RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1670525552843/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, i nternal error, NCCL version 2.14.3 ncclInternalError: Internal check failed. Last error: Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 1a000

    • @mnk_navin
      @mnk_navin Před rokem

      same error bro, did u rectified it?

  • @AmirHosseinSojoodi
    @AmirHosseinSojoodi Před rokem

    Thanks for the presentation. Please make MIG available for everyone

    • @pawelpomorski4025
      @pawelpomorski4025 Před rokem

      If you are interested in trying our experimental MIG setup on narval, please submit a ticket with a request. We might be able to give you some access in January.

  • @sinedeiras
    @sinedeiras Před rokem

    Dear Ge, thanks for the video. Dou you know if its possible to debug in many nodes ?

  • @userjjb
    @userjjb Před rokem

    The first example shown will not use multiple GPUs, only one. You can see at 27:36 that in the code (it would have been nice to have line numbers) [device = 'cuda:0'...] meaning it only uses the first GPU. I believe this should be [device ='cuda:0,1'...]

    • @lordnaive
      @lordnaive Před rokem

      u are correct

    • @kavibharathi1547
      @kavibharathi1547 Před 6 měsíci

      Can you tell exactly how to use or provide syntax to utilize multi gpu

  • @damakzb
    @damakzb Před rokem

    Thank you for the tutorial and very educational.

  • @TheRainHarvester
    @TheRainHarvester Před rokem

    Is there an openGL version of gnuPLOT? So others wouldn't need to install gnuPlot to din my program?

  • @whimsicalvibes6233
    @whimsicalvibes6233 Před rokem

    Thanks a lot.

  • @ranjeetkgupta
    @ranjeetkgupta Před 2 lety

    great talk ! Can we use Nsights for cluster wide profiling ?