Mozilla wpt-api Documentation

(a.k.a Using WebPageTest... Mozilla Edition)

Introduction

Project Goals:

Metrics & Metadata

Currently tracked metrics:

metric a.k.a. units type source
bytesInDoc total bytes num count Mult. sources[0]
domContentFlushed dcf ms duration timeToDOMContentFlushed - fetchStart in
wptagent/internal/js/page_data.js
fetchStart same ms, sec duration gecko
pageLoadTime pageload in Raptor ms, sec duration loadEventStart - fetchStart in
cloudops-deployment/projects/wpt/puppet/modules/wpt/templates/settings/custom_metrics/pageLoadTime.js.epp
timeToDOMContentFlushed TTDCF ms, ms duration gecko
timeToFirstByte TTFB ms, sec duration Mult. sources[0]
timeToContentfulPaint FCP ms, sec duration (1)user_pref("dom.performance.time_to_contentful_paint.enabled", true) in
wptagent/internal/support/Firefox/profile.prefs.js
and
(2)addTime("timeToContentfulPaint"); in
wptagent/internal/js/page_data.js
timeToFirstNonBlankPaint FNBP ms, sec duration user_pref("dom.performance.time.to_non_blank_paint", true) in
wptagent/internal/support/Firefox/profile/prefs.js
timeToFirstInteractive TTI ms, sec duration gecko
visualComplete visually complete ms, sec duration WebPageTest (video)
SpeedIndex Speed Index ... score WebPageTest (video)
requestsFull total requests num count Mult. sources[0]

[0] Multiple sources: (1) gecko (2) wptagent (3) another I'm forgetting? The key takeaway is that because WPT can't get a trustable set of network events from Firefox, it does log-culling/HAR-generating "magic" (less magic is good!) See https://github.com/WPO-Foundation/webpagetest/issues/1199

Test Metadata:

name type value(s) source
browser_version string "66.0a1" self.marionette.session.capabilities in wptagent/internal/firefox.py
browser_name string "Firefox Nightly" ...
### Manually Testing Metrics in Firefox
1. Enable/modify any (missing) prefs/pref-overrides
2. Open Tools -> Web Developer -> Web Console
3. Load a URL
4. After the page has "finished" loading, type something like: window.performance.timing.timeToFirstInteractive (DO take note of the other implemented/available metric options available as you autocomplete.)
5. Hit return/enter
6. You should see a value similar to 1542438605479 (time-stamped offset, in milliseconds)

PRO-TIPs: you can and should input window.performance.timing and/or performance.getEntriesByType("navigation") into the console, for the full data.

window.performance.timing:

twitch-performance-timing

performance.getEntriesByType("navigation"): nav-timing

Adding Metrics to WebPageTest with Firefox

  1. Manually test the metric + prefs; most metrics can be found in the following Firefox DOM WebIDL: mozilla-central/dom/webidl/PerformanceTiming.webidl
  2. If needed, add/modify Firefox's prefs.js, via a PR to wptagent/internal/support/Firefox/profile/prefs.js
  3. Example: https://github.com/WPO-Foundation/wptagent/pull/181/files#diff-69b0882d86377063fd0514c0dc978308
  4. Additionally, we might need to have the metric (if not available via standard APIs) emitted in WebPageTest, in wptagent/internal/js/page_data.js.
  5. Example: https://github.com/WPO-Foundation/wptagent/pull/230

Mini WebPageTest Compendium

You'll see something like the following JSON returned:

{
  "statusCode": 200,
  "statusText": "Ok",
  "data": {
    "testId": "190123_FA_dcbc2acb71e08a640f81999f9d2360dd",
    "ownerKey": "5f84816b2aaf6c795cb4dad85e2a472e004f58c2",
    "jsonUrl": "http://webpagetest.org/jsonResult.php?test=190123_FA_dcbc2acb71e08a640f81999f9d2360dd",
    "xmlUrl": "http://webpagetest.org/xmlResult/190123_FA_dcbc2acb71e08a640f81999f9d2360dd/",
    "userUrl": "http://webpagetest.org/result/190123_FA_dcbc2acb71e08a640f81999f9d2360dd/",
    "summaryCSV": "http://webpagetest.org/result/190123_FA_dcbc2acb71e08a640f81999f9d2360dd/page_data.csv",
    "detailCSV": "http://webpagetest.org/result/190123_FA_dcbc2acb71e08a640f81999f9d2360dd/requests.csv"
  }

To view the full test results in the Web UI, just use that userURL, which is http://webpagetest.org/result/190123_FA_dcbc2acb71e08a640f81999f9d2360dd/, in this case. * MOZ_LOG * networking * DNS * filmstrips * screenshots * optimization checks * timeouts * run_time_limit (180, sec) which is "Time limit for all steps in a single test run" in https://github.com/WPO-Foundation/webpagetest/blob/7b8d5d0821ae18b547f475133cae28a3c2b2778a/www/settings/settings.ini.sample#L60 * time (???, ???) which "Set[s] the timeout on a per-test basis (not documented because I was a bit worried about abuse but it's there)." found in https://www.webpagetest.org/forums/showthread.php?tid=3653&pid=25308#pid25308 * --timeout (120, sec) which is ": timeout for polling and waiting results [no timeout]" in https://github.com/marcelduran/webpagetest-api#test-works-for-test-command-only * timeout and time_limit** in https://github.com/WPO-Foundation/wptagent/blob/11222c7ab48bafb1203494dc4089fa298e75e040/internal/webpagetest.py#L429-L430 * maxtime(600, sec) which is "Maximum amount of time for a test run (if requested by timeout=X)" in https://github.com/mozilla-services/cloudops-deployment/blob/73ecc43a1c3a3da7c73a4d3d939b16e70cacf112/projects/wpt/puppet/modules/wpt/templates/settings/settings.ini.epp#L23-L24 * max_run_minutes (60, min) which is "Force individual runs to end if they didn't complete." in https://github.com/WPO-Foundation/webpagetest/blob/7b8d5d0821ae18b547f475133cae28a3c2b2778a/www/settings/settings.ini.sample#L63 Also see https://github.com/WPO-Foundation/webpagetest/commit/ae11833a986260cf83f66b10fff4a9648f8dfa23, which added it * step_timeout (120, sec) which is "Default timeout for each step of a test (in seconds)" in https://github.com/WPO-Foundation/webpagetest/blob/7b8d5d0821ae18b547f475133cae28a3c2b2778a/www/settings/settings.ini.sample#L54 * video * webPageReplay

webpagetest

wptagent

Infrastructure

WebPageTest Instance (Mozilla-internal)

To run a test via the API, using our internal instance at wpt-api.stage.mozaws.net, is something like:

$ webpagetest test www.twitch.tv  --server https://username:passkey@wpt-api.stage.mozaws.net --location us-east-1-linux:Firefox --bodies --keepua -r 3 --first --reporter json

To run the same test, but using the public instance now at www.webpagetest.org, would be:

$ webpagetest test www.twitch.tv -k your_API_key --location ec2-us-east-1:Firefox --bodies --keepua -r 3 --first --reporter json

Jenkins

AWS EC2 Setup

Dependencies

Core:

W3C (Draft) Specs


Contributions

psst, this could be you!

Credits & Thanks