Research & High Performance Computing - Computerphile
Vložit
- čas přidán 29. 06. 2024
- A supersized game of tetris - Dr Jim Wilson on scheduling High Performance Computing jobs and helping people get the best out of Nottingham's HPC.
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
Nice of Computerphile to blur out the handwritten passwords on those machines. Very thoughtful.
Sarcasm?
You really wonder why they chose that spot to film in the first place.
Yes thoughtful, because otherwise they would be very careless if they displayed those to the public. It’s not very uncommon in labs, we personally keep user/pass for software right on the computer of most of our analytical equipment.
+H Kr yeah, bit of a mistake, assumed they were decommissioned machines then the labels were pointed out at the end... >Sean
Computerphile Blurring to censor writing is not a good idea. Deblurring technology is a thing, and it's more powerful than you might expect. I wouldn't be surprised if you haven't blurred it enough to prevent someone from being able to read it.
If you really want to make sure something can't be seen, you need to cover it up with a solid color.
this dude is high level
I don't know why but this guy is genuinely the best person I wachted talking on CZcams in 2017 plus the bit for 2018 so far.
Well done. You should get into the problem of parallelization a bit more. Non-techies don't come in with already parallel algorithms so getting their research goals to run on 1000 cores can be interesting. And in general, even for the best techies, parallelization can be difficult at times.
This is very true. Just take a shot at writing a GPU program to solve a parallel solution to a problem. Even when the problem and potential solutions are known, it quickly spirals into a bizarrely fun nightmare trying to come up with a sensible way to distribute the load for maximum efficiency.
I think they could have gone more into the "jigsaw" aspect as well. Scheduling the jobs is like a 3d box packing problem. They showed time and cores, but there's memory requirements too.
My OS class had a guest lecture from a guy from MS Azure. The scheduling algorithms for maintaining high utilization of their machine resources are pretty complex.
This is great content. I would like to see more videos focused on this area.
Working in HPC is awesome. You are by definition years ahead of everybody else. The problem is this also means nobody yet understands the problems you run in to :)
I work as a software engineer in the semiconductor industry and the physicist we work together with knows more about (HP)C than our whole team combined (we only write regular software ourselves). It's quite weird to be outclassed like that on your own area of expertise.
That choice is based on the price tag only. Not competence. And your racism is uncalled for.
+ashutosh Mishra What an ignorant comment.
@ashutosh Mishra if you don't mind explaining, can you please explain your opinion as to why or what makes your statement true. Is the Indian emphasis on education far superior? Subjects offered? age level education benchmarking? Thanks
Back to the Computerphile we know and love - lots of images of blinking lights and big machines that go bing AND we can hear the interview 👍😎
7:39 "spraying stuff all over the room, which i have also done" this quote is fun out of context xD
I'm a computational physicist and I support this video.
I think you should do more videos on parallel programming. It's a very relevant topic at the moment and there's quite a lot to cover, like programming for multiple cores, GPUs, and multiple processors (as in the case of this video). Not to mention processors, threads, scheduling, and all that jazz.
Really interesting stuff.
As a graduation student in System Analysis, I need some more info on this topic to move on and get a good grade, and a lot of other channels don't seem to explain that well as this one.
Would love to hear more about any recent algorithmic development in HPC - after a lot of early hype about ways to get around the surface to volume (data to computation) it sounds from this interview that most problems can still only take advantage of HPC if they are “trivially parallelizable”, i.e. a bunch of completely independent processes that don’t share data, and that HPC in this case is just fancy job distribution book keeping.
good job on bluring out the labels out on the pc"s next to the professor ive seen amature people bluring out stuff and ending up screwing up for one frame and reveling the info
Hire a game theorist to make the sharing of resources feel fair to everyone. :)
actually there are algorithms for scheduling jobs in parallel
Those lovely core 2 and Vista stickers. Bleeding edge tech to be sure.
Love this guy
It reminds me to that part on the shin Godzilla movie where the japanese found out a way to halt Godzilla's cells but had to figure out the appropriate active molecules to dock with Godzilla's receptors, so they asked Germany (if I recall correctly) unlimited use of their HPCs . I'm glad to see that writers handled these kinds of details
if you get a camera stand, you can get way more stability during low-movement segments like most of this video.
Think about what (if) could come out of "daap supercomputers" (off the top of my head, currently the ones in testnets are Golem, SONM, and partly iExec) in terms of scientific research, especially medicine.
That doesn’t sound like “high performance” so much as it just sounds like “a lot”. How much does he work with scientists to perfectly optimize their programs to run as quickly as it can on the hardware?
Dam this guy is what I dream to be. I'm completing my bachelor's in chemistry right now and just did a summer internship in high performance computing at a national laboratory. Before this, I thought I was somewhat knowledgeable about computer science but now it feels like I just know about 1%. Found his university bio but would be interested in to know that apart from chemistry what education/certifications/trainings did he do in HPC? I assume initial hpc knowledge could be self taught as there's ton of information online about it and then forming your own project with hpc
I wish to know something. I run some experiments on my Raspberry Pi server 24/7 since it's always on, unlike my main PC. I design them so after a week or two I can read a file and see what it came up with so far without interrupting. Is there a possibility for a researcher to run an HPC task for unspecified period of time and see the results in the meantime? And re-run the task with different parameters if the results went in the wrong direction?
I imagine it depends on the software being used, and what task is being executed. My experience is with Monte Carlo radiation transport codes which will yield an approximate result in a (relatively) short length of time, but improve accuracy with simulation length. In this case it's trivial to set an indefinite simulation length and configure the code to dump periodic output files to monitor progress, but I am sure there's software where this would not be viable due to the nature of the task.
Do they have any Threadripper mahcines?
What is it behind that blurry censors?
In the biology field for testing receptor sites in molecules and cells, I really hope they do not just test for the intended target site, but also at the same time every other site the same molecule could attach to. That way they can find out possible side effect areas beforehand.
Another problem could be medicine chirality. Test it in a parallell simulation before developing the drug only to find out afterwards it ALSO attached to a not intended target site somewhere else in the body. Or as in the case of "the pill" it ends up in frogs or crocodiles and make them female thereby destroying whole species because of unintended consequences.
Test 1: Will it hit target?
Test 2: What else will it probably hit?
Test 3: Can we hide our cryptomining operations to the tax payer and server park manager?
Result: Retire early will billions of bits in the ba, in the vau, on your harddrive.
Would it be possible to simulate such a computing job? From designing a simple abitrary problem, to implement it into the cluster and see the result?
They already do this.
4,000 core days seems like a pretty generous limit, at least until my job gets stuck on a single-core node for 4,000 days.
"The answer isn't wrong. The problem is that you didn't ask the right question" ~ Deep Thought
I'd love to work as a tech in such an interesting facility. I've got loads of datacenter, computing and linux experience. Gib me job pls :).
Surely this drug molecule matching he is referring to will just be calculated once and then saved to a database for further reference, hopefully a public database.
Professor Moriarty must get a lot of Sherlock Holmes jokes.
Do you know if it Is possible to achieve a cluster setup that ultimately has a normal Windows 10 User Experience for one node, But incorporating the processing power of multiple Computers.
Say I have 4 Computers that are all relatively similar in performance networked on 1GB LAN and one of those is my computer that I run applications like Adobe - Blender - Fusion360 - AutoCAD Normal workstation applications in an ordinary Windows 10 environment, Is it possible to combine these other unused computers to increase productivity of my workstation?
Currently having to move files around , have multiple installs of applications on multiple PC's and using VNC to interface with them, it really is incredibly cumbersome for 2021!
I would much prefer a regular desktop experience with added benefit of the combined power instead of wasting all this compute power by having it used inefficiently or not at all.
If so could you produce a tutorial, I would think it would become popular video in any case :)
Thanks in advance and cheers.
When I grow up im gonna be a HPC
OT: Funny, people seem to have real trouble pronouncing azure. I heard Escher, Asia and now ad-sure. All wrong. The German pronunciation (I'm from Austria) is also wrong, but as far as I can tell out of these still the closest (at least to my Austrian ear). Written in phonetic German it should be Azür, I think.
- What are all those "stone" machines for?
- Duh, looking for drugs!
Well there’s just such a demand for high quantum computers but there’s just not enough use for these type of technology
This*
"which I've also done" lol
But.. Will it run crysis?
If I had a dollar each time I read a comment like this I would be able to afford rent.
Nidhal25 ❤️❤️❤️❤️
I bet it could run millions(thousands maybe more down to earth) of crysis instances at once
But.. can it run witcher 2 on uber mode? Seriously is there a computer that can do this?
those which can run crysis do just that :)
"equivelant to the professor having an explosion in the lab, ..., which i've also done" - Totally inept professor /s
Who hasn't in their career? Most get that particular achievement (in one way or another) under the belt by the time they're doing their bachelor thesis.
He reminds me of a middle-aged, less hairy Kitt Harrington.
your tetris is broken. tetris pieces are always four blocks.
HPC and performance evaluation: two things which don't go well at all. HPC is great for researchers outside of Computer Science, but not for software developers.
128.245... ohhhh cant quite read it!!!
Apperently CZcams notifications brought me here before the first like.
I mean, what is even this timespan, sometimes it pops up a few seconds after the video is out.
And sometimes it literally only tells me about new video 30 mins later
And thats how you write a fancy "first" comment
+Jesus of suburbia it often doesn't tell me for days, and I press the button - Sean
U know how sad it is when your comment didn't get responded to, but then a guy just copy-pastes it on another vid and the author answers?
(Even tho it wasn't supposed to have an answer in the first place)
Early adventures
Much fun
I completely understand your feelings, I was a key player in the CZcams creative comments business, but couldn't achieve much success and fame. So yesterday I decided to become evil and start plagiarizing. This is my first project as a plagiarizer, please support me by liking.
What makes u think otherwise?
"Docking" tehe XD
Stop drop and mooc there's a guy that real
XD
HPC? Vat da faak?!
High Performance Computer
Or Hydraulic Press Channel ...
Jonas got it :P
Bioinformatics!
"Droog"... that's a peculiar way of pronouncing "drug". Reminds me of The Clockwork Orange.
Nikolay Yakimov Northern English accent
How is it even slightly odd. How else do you say the ug sound...
One of the Lancashire accents I guess. He's using the 'u' vowel though....definitely not 'o'.
Not Lancashire but somewhere in the North East (not sure exactly where) >Sean
grozny bratchnies...
Sentence.
Also, you cant just copy a comment from the previous vid and copy-paste it
Dats rude
Sorry bro, I was drunk.
Scientific Linux is based on Red Hat. Noo.
bitcoin
Pretty sure if Brits didn't waste time and energy on things like how they pronounce 'H' we wouldn't have won our independence. "Haych" sounds ridiculously laborious for such a small thing. It's practically two syllables.
But I like the fricative H :(
Speaking as a Brit, I hate the "Haych". I have always pronounced it (and spelled it) as "Aitch", as do most of the people I know.
That's only particular parts of the UK, such as the NW of England. It's also how we pronounce it in Ireland. Nothing cumbersome about it, given you don't end up pronouncing a glottal stop at the onset.
*succ*
‘Docking’ lol
2:28 Hello! I'm from the Internet. I'd advise you to explain what something is before announcing its name. #domainSpecificTerms #YouAreOnTheInternet #BOINC
He explained it right after saying it
Go home internet. You're drunk.
He explains it right after...
Yaz he specifically stated that it should be stated prior to the name being given... your point is irelavent regarding the original comment; you're discussing something else entirely.
Casper S? He specifically stated that it should be stated prior to the name being given... your point is irelavent regarding the original comment; you're discussing something else entirely.
This is the reason why I hate crypto currency mining.
All those wasted cpu cycles that could have been used for this type of research.
Money over human life.
Praising of big pharma 😒
That is if you live in a first world country, have insurance and enough personal finance to pay for it that is.... Yeah great health as a luxury product what an amazing bunch of people; pay or die~
Dantali0n what do you expect actually? for people to give you medicine and consultation for free? and where will they get money for research, salary for those who have helped you or simply running a pharmacy store? that would be possible if you paid extremely high taxes but i assume you wouldn't like that either
capitalism is your enemy, not "big pharma", stop with the nonsense, it's dangerous and we don't need more antivaxxers and such
Suvi-Tuuli Allan actually bad capitalism is the problem, capitalism itself is essential for our society to work properly (because of many people with various opinions and needs), you couldn't possibly make it work for everyone or at least a fair number of people without it. but if you have any ideas how you could do it, I'm open, please present them
these responders are so naive.. probably aren't even aware of the conflicts of interest or that 'big oil' took over the education system to shape the medical industry into what it is today, a cesspool of fraud.