I'm loving all these updates and improvements you're doing, I'm excited to see this service working! I hope it could have an online trace viewer, so we can check the results without needing to upload the .zip, which could be an interesting upgrade and differentiator versus the other cloud solutions.
Is there going to be a better integration with Azure DevOps? I've connected our Azure DevOps to the Playwright Testing resource. I'm able to run my tests there but the reporting is terrible. To not be able to see the video or traces within the pipeline run itself is a serious limitation. Do you know if Microsoft is working on that?
So basically you can run tests in browsers that are not on your machine, so tests can run much faster if I understand properly. Is it all free? I can imagine that letting all users use these browsers can be very demanding, so I assume that it is not all that free, right?
I assume we need to have stable test environment to run 40 tests at once? With this solution we are removing part of the load from the CI but still environment might be a bottleneck, correct?
Excellent question! It's important to make sure your target application's environment can manage the number of concurrent requests you plan to test. This requirement varies depending on your application's characteristics, the aspects your tests are focusing on, and potential network delays. To determine the most effective number of workers for your specific scenario, we suggest experimenting with different quantities to find the ideal balance for your use case. Often, you may find that running fewer parallel workers can still yield significant performance improvements.
3 questions 1. I checked, it seems Southeast Asia not supported. Any idea when this will be released to Southeast Asia? 2. If my project has 100 tests than run for 10 mins, are those considered as 1 test for 10 mins duration? wonder how the pricing works. 3. Can I use playwright to run test on self hosted server (instead of your service)?
I wonder you have the browser instances somewhere in the cloud. How will they reach internal applications that are not accessible via outside connections. Because now we run playwright on our own internal azure devops runners or our local machine but i dont see how cloud browser will have access to our internal app.
I was exploring playwright java and loved it. But I wanted to make use of Python's flexibility so switched to Python playwright. But there I am struggling a bit to understand how I use asyncio properly. There is a sync_api and async_api and both of them run async. Wish if someone could guide me to a better documentation. I am not planning to use JavaScript. I will stick with Java or Python.
It supports the same browser versions like the Playwright project is using. See the release notes for more details. Microsoft Edge is also supported. File an issue here for more questions: github.com/microsoft/playwright-testing-service
Yes, bypassCSP will just work! can you post your question on the discord server for the service: discord.com/channels/807756831384403968/1161681421774094366
I could be wrong this is more about running your tests, would still develop them locally, and when a PR and a test pass are needed... Would be the same as devs working on product code, hard to see debugging in a remote web environment whey they can debug local?
As per above yes. Debugging strategy is the same. We recommend traces on CI and UI Mode / VSCode extension locally. File an issue here for more questions github.com/microsoft/playwright-testing-service
It should work! Can you report this on the discord service so the azure team can follow up: discord.com/channels/807756831384403968/1161681421774094366
I'm loving all these updates and improvements you're doing, I'm excited to see this service working! I hope it could have an online trace viewer, so we can check the results without needing to upload the .zip, which could be an interesting upgrade and differentiator versus the other cloud solutions.
Is there going to be a better integration with Azure DevOps? I've connected our Azure DevOps to the Playwright Testing resource. I'm able to run my tests there but the reporting is terrible. To not be able to see the video or traces within the pipeline run itself is a serious limitation. Do you know if Microsoft is working on that?
Hello?
This is amazing !
Will the Public Preview be coming to the UK regions?
So basically you can run tests in browsers that are not on your machine, so tests can run much faster if I understand properly. Is it all free? I can imagine that letting all users use these browsers can be very demanding, so I assume that it is not all that free, right?
The first 100 test minutes are free in the 30-day trial. Pricing details are here: aka.ms/mpt/pricing.
Wow! Looks great! What about free tier plan? How much it covers?
The first 100 test minutes are free in the 30-day trial. Pricing details are here: aka.ms/mpt/pricing.
This is cool but, is it support Safari too?
Yes it supports all Playwright browsers. File an issue here for more questions: github.com/microsoft/playwright-testing-service
I assume we need to have stable test environment to run 40 tests at once? With this solution we are removing part of the load from the CI but still environment might be a bottleneck, correct?
Excellent question! It's important to make sure your target application's environment can manage the number of concurrent requests you plan to test. This requirement varies depending on your application's characteristics, the aspects your tests are focusing on, and potential network delays. To determine the most effective number of workers for your specific scenario, we suggest experimenting with different quantities to find the ideal balance for your use case. Often, you may find that running fewer parallel workers can still yield significant performance improvements.
How’s this any different than what browserstack or similar offers? Also - what’s the max number of workers you support?
It is working fine for typescript tests. But not for .net c# tests. Can you please tell how to run playwright tests created in c# in Azure cloud?
3 questions
1. I checked, it seems Southeast Asia not supported. Any idea when this will be released to Southeast Asia?
2. If my project has 100 tests than run for 10 mins, are those considered as 1 test for 10 mins duration? wonder how the pricing works.
3. Can I use playwright to run test on self hosted server (instead of your service)?
i love it! :)
It's interesting to know how to integrate for languages like java or python
So far it’s only available for Node.js, stay tuned!
@@Playwrightdev can we expect this feature to be available for dotnet in the future?
I wonder you have the browser instances somewhere in the cloud. How will they reach internal applications that are not accessible via outside connections.
Because now we run playwright on our own internal azure devops runners or our local machine but i dont see how cloud browser will have access to our internal app.
We provide a feature to expose your localhost. See here: learn.microsoft.com/en-us/azure/playwright-testing/how-to-test-local-applications
Cool feature ❤
Love it!
I was exploring playwright java and loved it. But I wanted to make use of Python's flexibility so switched to Python playwright. But there I am struggling a bit to understand how I use asyncio properly. There is a sync_api and async_api and both of them run async. Wish if someone could guide me to a better documentation. I am not planning to use JavaScript. I will stick with Java or Python.
check this page in docs which might help: playwright.dev/python/docs/library#usage for any other questions please use our discord server
Does the playwright service support a specific browser versions. Also can we run against Edge browser as well instead of Chromium?
It supports the same browser versions like the Playwright project is using. See the release notes for more details. Microsoft Edge is also supported. File an issue here for more questions: github.com/microsoft/playwright-testing-service
Nice!
Awesome. What about the test reports?
please use our discord server for all questions regarding the service: discord.com/channels/807756831384403968/1161681421774094366
Discord is blocked at our company. What other options do you have?@@Playwrightdev
I see. you can use GitHub issues / feature requests: github.com/microsoft/playwright-testing-service
Can it bypass CORS for Firefox and Safari?
Yes, bypassCSP will just work! can you post your question on the discord server for the service: discord.com/channels/807756831384403968/1161681421774094366
What about debugging such tests?
I could be wrong this is more about running your tests, would still develop them locally, and when a PR and a test pass are needed... Would be the same as devs working on product code, hard to see debugging in a remote web environment whey they can debug local?
As per above yes. Debugging strategy is the same. We recommend traces on CI and UI Mode / VSCode extension locally.
File an issue here for more questions github.com/microsoft/playwright-testing-service
doesn't work for API Tests. It says "No tests" found if you dont use a browser
It should work! Can you report this on the discord service so the azure team can follow up: discord.com/channels/807756831384403968/1161681421774094366
Where is Andrey?
How to make it work if we are using cucumber runner instead of Playwright runner?
Can you post your question on the discord server for the service: discord.com/channels/807756831384403968/1161681421774094366
Looks like this will unemploy a number of people....? If Debbie won't need to get 6 coffees while her tests runs.... oh well...
So I have to pay money to run test ?. Nah =))
only if you want to scale so that running hundreds of tests takes less time. But to run tests normally and in parallel there is no cost at all.