AlphaGo Zero vs. Master with Michael Redmond 9p: Game 2
Vložit
- čas přidán 30. 11. 2017
- Michael Redmond 9p, hosted by the AGA E-Journal's Chris Garlock, reviews the second game of the new AlphaGo Zero vs. Master series. Download the sgf file here: www.usgo.org/news/2017/12/alph...
"Although the openings in this series are pretty repetitive, the games themselves vary," says Redmond. "So in some, you'll see a half-point game, and in others we'll see Master crash. This game is interesting because it's the first time that Zero has black. Also, later in the game, I get the feeling that Master is acting like it did in the 60-game series earlier this year against top human players, where it thinks its winning and is sort of closing up shop and wrapping up the game. So I wonder whether it mis-read a tsume-go -- actually a 60-move sequence -- in this game."
Video produced by Michael Wanek and Andrew Jackson. The sgf files were created by Redmond, with editing and transcription by Garlock and Myron Souris. - Hry
Wow, Zero's strength is just incredible! Michael's identification of the set up move to eventually capture the top right is astonishing. It's really humbling for a human player to see such a long readout and realize what Zero just did there. The truly awe-inspiring thing is that Zero probably had several other potentially strong uses for that move that we never actually got to see in this game. These analyses sometimes just leave me speechless from pure gratitude.
Zero is 13 dan!
This is so rich. Thank you Michael, Chris and the AGA for making these.
Truly unbelievable Go. The stone on S13 shows just how deep and dynamic the game can be when its played at the absolute highest level.
Thank you again for your time. Very instructive commentaries. I may never be a Go player myself, I'm terrible seeing where to play next, but I do appreciate watching these videos, freely available, like stories between giants.
Anyone can play go! The rules of the game are very simple and easy to learn. Getting good at the game is another thing entirely, but I think you can enjoy the game at any level. If you pop on to a server like OGS or KGS I am sure that you'll find people willing to teach you aswell as opponents of your level.
I went to a go tournament once and lost to a 6 year old. If you want to play against me let me know. LOL
I believe the game that Michael Redmond refers to is the 1861 castle game between Honinbo Shuwa and Matsumoto Inseki. Honinbo Shuwa was the greater player and was hoping to make his case to become Meijin godokoro. However, in this game he lost by a single point and lost hope of ever becoming the Meijin. Mastumoto was so excited for his successful challenge against his go house's main rival that he made hundreds of copies of the game and distributed it to anyone interested (and maybe some people not so interested as well!) Hope this helps.
Michael repeatedly talks about the difference in Monte-Carlo algorithm in Master vs Zero. Let me give some additional info on the difference between two algorithms.
The traditional Monte-Carlo algorithm consists of two more or less independent parts. First one, is what's called Monte-Carlo Tree Search, or MCTS is an algorithm that decides which moves to explore. It favors the better moves, but also evaluates some moves that look worse. The idea is to spend the majority of the reading resources around the main variation, but also probe various other moves to try and find some unexpected "trick-moves".
The second part is to use the playouts to evaluate the position, that is in a leaf of the tree. A semi-random game with a certain policy is played, starting from this position, and its result is used to evaluate the position.
AlphaGo Master uses both parts of this algorithm, while AlphaGo Zero only uses the the MCTS and then the value network to evaluate the position. This results in less randomness than with Master, but not by much, since the expansion of the game tree in MCTS is also semi-random.
Finally, one property that affects the choice of the moves is the time control. With longer games the same algorithm will become much more predictable with its choice of moves, which might be the case for Master, which played much more varying moves in the Master series with with 5 seconds per turn, but plays essentially the same opening with the longer time control.
well... i was under a different understanding:
The original alphago used two networks. The first to trim the tree of the search to favor "pro-like" moves, which are identified by the feelings of the neural net, and executed upon en masse with some variety and some backpropogation reweighting the branches (the MCTS). The second network is used to evaluate the goodness of the search's position at some midpoint in the search (so it needn't permute all the way to the leafs) again by the feelings of the neural net. Just as you've pointed out.
AlphaGo Master improved on the original by adjusting the favoring of the moves during the tree-trimming phase through self-play. Naturally, the second neural network improves in evaluating positions when fed better games. But, Master still functioned as the original. See game position -> play out moves, trimmed -> evaluate positions -> suggest best move.
AlphaGo Zero is a frickin' mystery. *It doesn't use MCTS to play!* yah. true story. MCTS is used to inform the policy network during self-play as an improvement operator. Then, *Zero plays entirely from policy network*. I don't really know how that's possible, but like, it's just one of those things that makes zero epic. It doesn't require master's computational volume once trained. That's why it can play on so much less resources, too. The amazing part of Zero is how in the hell did deepmind inform the policy network so well? "New reinforcement learning techniques" my ass!
+Jeffrey Kane, that’s not quite what happened. First of all, all the versions of AlphaGo, including the first one, that played with Fan Hui, used self-play to improve.
The version of AlphaGo described in the first paper started by training the policy network to predict the pro’s moves, and then improved it via self-play. It also had a value network that was trained to predict the outcome from playouts of policy network. During the play, it explored the position tree via MCTS algorithm and then used the combination of playouts and value network to evaluate the position.
I would guess that the version playing with Lee Sedol was similar to the one, described in the first paper. The Master & Ke Jie version(s) were much further down line, and we don’t have any in-depth paper that would describe their configuration. My guess is that they use the algorithm closer to the one of Zero, but still seeded with human games, or at least games from the previous versions of AlphaGo.
Now AlphaGo Zero technically has just one network, but since it outputs both the evaluation and the probabilities of moves, it works as if it were two separate networks, for value and policy. As I wrote in my first comment, they completely removed the playouts, but they still do the MCTS search.
They actually did try to generate the moves just from policy network, and the resulting version was fairly strong, but still 1500-2000 Elo points weaker than the same network with MCTS.
Source: I actually read both papers, and implemented (simple) MCTS for my pet project.
i guess i'm misunderstanding .. the playouts ARE the MCTS .. like, that's what mcts is. How can they remove the playouts and still do mcts? It was my impression that zero only used mcts in training as a kind of gradient descent algorithm.
See my first comment. There's more than one part to it. Have a look at the Wikipedia page: en.wikipedia.org/wiki/Monte_Carlo_tree_search, but replace the "Simulation" step with evaluation by a value network.
right... but the "monte carlo" part is the simulation.
Thanks a lot for the video. We're really lucky to have these weekly releases :)
You make such an amazing duo. Thank you for all your work. Since AlphaGo vs Lee Sedol, it has been such a bliss to see all your co-presentations :) Keep up the good work.
Thank you so much Michael and Chris! I can’t tell you how excited I get on Friday mornings waiting for a new video to come out in this wonderful series. You are both fantastic. (I will find out if it’s possible to join AGA from Canada, and will do so if yes).
Thank you Chris and Michael :) These are always wonderful. No such thing as too many variations.
Super thorough explanations, allowing even someone like me to follow this high-level play. Thanks Michael and Chris!
Thank you for making these.
I was about to go to bed and then I saw this video. Friday night is well spent. Thank you, Chris and Michael.
Fantastic job guys. Michael's great readouts and Chris's responses are much appreciated. This really is the pinnacle of the sport.
Thanks guys, I really appreciate it.
Very insightful analysis by Michael.
These are great! So far I have liked the zero-master games better than the AG self play games. Also just an extra shoutout to the amazing amount of work Michael Redmond has put into this.
The invasion point Q18 was not so hard to guess... S13 though, wow! and brilliant analysis on those moves. S13 shows the kind of staggering foresight that I expect from AGZ, but without Michael's commentary we would not be able to appreciate it!
And thanks for the good advice on the heterodox cut in the lower left. As Miss Marple(!) said "people generally ignore good advice, but that is no reason not to give it". Good to know Michael appreciates her wisdom! :-)
As for epic first-line crawls, there's also the first game of the Go-Kitani Kamakura match.
Is the link of the sgf available already? Anybody knows about the classic game of crawling on the first line mentioned by Redmond?
Great commentary. Michael, are you working with other Professionals in Japan in analyzing these games in a study group or something? I hope you are not totally isolated here. Maybe you could Talk to Fan Hui to get some information about these games.
What I would LOVE to see is how Alphago would get on with the problems in Shuko's "The Only Move" books. I was looking at a couple of those problems where the answer was a stylish thick move recently, thinking "Alphago would surely not play there!" - and yet, perhaps that would not prove that Shuko's move was incorrect...
Thanks for the great video! I do prefer the earlier pronunciation of Ke Jie.
Clearly Master had Zero chance once the...
*cough*
Anyway, it would be really interesting to see what would happen if Master and Zero trained against each other for a while. Zero seems to keep hitting Master's blind spots. I think they'd both end up significantly stronger.
I love watching these videos, but I can't support you and join AGA, because I'm not from the US :(
5:10 I will have to stop you right there Michael!
I actually thought you finally got it pretty good in the last couple of videos. Please continue pronouncing it the way you were doing so! Haha.
To someone not familiar with Chinese pronunciation, it may not sound that strange, but to those that are, saying it as "Kay Jay" sounds really awkward.
I think of it as the initials KJ, in which case it's not wrong.
Mets That still doesn't quite work. Imagine Michael saying the "LSD version" or perhaps the "FH version".
ConsciousBreaks - I think he’s just saying the initials to avoid having to pronounce the name.
Please refer to my above comment.
ConsciousBreaks - What do you mean it doesn’t “work”? He doesn’t have a problem pronouncing Lee Sedol or Fan Hui. He _does_ have a problem pronouncing Ke Jie. That’s why for Ke Jie, he just says the initials K J. What is so complicated about this?
柯洁。
Tip for pronouncing Ke Jie: The first vowel sound similar to that in "good"; the second is like "jet".
If you click the speaker icon in Google Translate here you can hear it:
translate.google.co.uk/#zh-CN/en/%E6%9F%AF%E6%B4%81
I mean sure, but I don't think that's the best way to describe it.
Sure it is, unless you're going to use IPA (which non linguists will not understand) and be properly exact. Even then the English vowels I provided are very close, and the reality is that most people don't speak Chinese and can't tell the difference. And there's a link to hear the actual sound; how you can describe it better than listening to the actual pronunciation?? I noticed you didn't provide an alternative way to describe the sound; care to elucidate us on the best way?
I was just saying that I was doubtful that someone who is uninitiated wouldn't quite get the correct pronunciation from that. My primary way of explaining it on the internet is to write "kuh jee-yeh" although it doesn't tell how to do a proper j sound. I like that you put a sound clip, but listening to that with an untrained ear can easily screw that up a bit more than having a written form.
That's worse though. Who knows what "kuh" is supposed to sound like, and I'm certain most would guess you meant the sound from "fun"/"huh" which is completely wrong. This is why I use actual words to make sure I'm conveying the vowel sound I intended (assuming an American accent, of course). But even if they guess the sound from "could"/"good", it's still not the same as the vowel sound in Chinese. And yes, J has a different pronunciation too. Furthermore, even if they get the vowels and consonants right, you _still_ have the problem of tones. Ultimately we just have to accept that most people are going to pronounce these with an American accent, and given some of the terrible pronunciations I've heard like /kɛ dʒi/ ("keh" with the vowel from "get", "jee" as in "jeep") that are nowhere near correct, I'm happy enough to just hear a passable English approximation!
I couldn't think of a good English word for the vowel in "洁", but I figured people would get the sort of Y sound following the J from listening to the sound. Maybe it wasn't a good idea for me to assume that. Since then I've thought of "Kiev"; do you know a better one?
So I think a better way to indicate it than what I first wrote might be, "say the words 'cou(ld) je(t)', but omit the letters in parentheses and make the vowel in 'jet' more like 'Kiev'". What do you think?
I guess that is more accurate, but I guess we're reaching the point we're it's neither simple nor accurate, which ruins the entire point of providing an example.
I do like the idea of using the vowel sound from "Kiev," it seems to be a close approximation.
But I think how Redmond used to pronounce it was enough to make me thank goodness that he didn't say it like Dwyrin.
Just one question
Why does a very strong player like master plays an opening which he considers bad? (The top left 3-3 joseki i mean)
I think even randomness does not cause Master to choose this, because as far as we know ai thinks it is really bad.
I don't think H9 in 20:17 is a good move.
First, white gives up the sente, and H9 does little help to the upper side.
Second, white doesn't catch the stone in J4, and black L4 will be the second largest move. (The largest move is the upper side.) So black will play the upper side first and wait for the opportunity to play L4.
I also agree that G8 in this game is not a good move either, so I think K5 will be the best move in this situation. At least K5 completely catch J4 and it doesn't strengthen black left side at the same time.
+
You know nothing.
I just know what I know.
You don't know what you don't know.
Show me your rank. I'm 9 dan in most of the go server, including tygem and fox go. Maybe you could teach me how ignorant I were.