Go-DOM - major milestone
After just under 2 weeks of work; I finally reached the first major milestone for Go-DOM.
Now, the browser will download and execute remote JavaScript when building the DOM tree.
A Brief History
The project started as a crazy idea; seeing that Go and HTMX is a stack that is gaining some popularity;
Go already has all the tools you need to test pure server-side rendering. But when adding a library like HTMX, the behaviour of the app is the result of a choreography between the initial DOM; the attributes on interactive elements; the HTTP endpoints reached, and the content delivered by those endpoints; both response headers and body. To verify the behaviour from the user's point of view, you need a browser; or at least a test harness that behaves ... not entirely unlike a browser.1
Searching for "headless browser in go" only led to results suggesting to use a real browser in headless mode. This combination that has a huge overhead; discouraging the fast and efficient TDD loop. Relying on a real browser will typically slow you down instead of speed you up.2
So the idea was sparked; write a headless browser in pure Go as a testing tool for web applications;
The first uncertainties to address was the parsing of HTML; as well as script execution. I managed to quite quickly; within 2 days to address both of these. I had a very rudimentary HTML parser; as well as I had integrated v8 into the code base3 and make Go objects accessible to JavaScript code.
The HTML parser was later removed, as go x/net/html already implements an HTML parser; dealing with all the quirks of HTML parsing. Parsing a well-formed document isn't a terribly difficult problem to solve. It's gracefully dealing with malformed HTML where it becomes tricky.
After some time, I also managed to get inline script execution to run at the right moment; i.e. the script is executes when the element is actually connected to the DOM, not after the full HTML has been parsed.
Working with HTTP requests
After being able to process an HTML document with inline script; the next step was to actually download scripts from source. This needed to integrate an HTTP layer; such that the browser fetches content itself; rather than being fed content.
The http.Client also allows you to control the actual transport layer using the http.RoundTripper interface. Normally you start a server; which will listen for requests on a TCP port. In this context TCP serves as the transport layer; but is itself irrelevant for the processing of HTTP requests. Due to simplicity of the standard HTTP stack in Go; an entire HTTP server is represented by a single function, func Handle(http.ResponseWriter, *http.Request).
The headless browser can completely bypass the overhead of the TCP stack and call this function directly using a custom RoundTripper.
Now the browser can perform HTTP requests, but the browser code itself is ignorant of the fact that the HTTP layer is bypassed. And with this came the ability to download the script during DOM parsing.
Example Code
Let's explore a simple test, as it looks now in the code base (the code uses Ginkgo and Gomega, a somewhat overlooked combination IMHO)
First, the test creates a simple HTTP handler which serves two endpoints, /index.html, and /js/script.js.
It("Should download and execute script from script tags", func() {
// Setup a server with test content
server := http.NewServeMux()
server.HandleFunc(
"GET /index.html",
func(res http.ResponseWriter, req *http.Request) {
res.Write(
[]byte(
`<html><head><script src="/js/script.js"></script></head><body>Hello, World!</body>`,
),
)
},
)
server.HandleFunc(
"GET /js/script.js",
func(res http.ResponseWriter, req *http.Request) {
res.Header().Add("Content-Type", "text/javascript")
res.Write([]byte(`var scriptLoaded = true`))
},
)
// ...
The intention here is merely to verify that the script is executed. To do that, the script produces an observable side effect: It sets a value in global scope.
To verify that the script was executed is just a matter of examining the global scope, which is done by executing ad-hoc JavaScript from the test itself; verifying the outcome of the expression.
The code to create the browser, load the index file, and verify the observed side effect
browser := ctx.NewBrowserFromHandler(server)
Expect(browser.OpenWindow("/index.html")).Error().ToNot(HaveOccurred())
Expect(ctx.RunTestScript("window.scriptLoaded")).To(BeTrue())
Test execution is also pretty fast. The part of the test suite involving JavaScript execution currently consists of 32 tests running in 23 milliseconds.
Next milestone, integrate HTMX.
As the project was initially conceived while trying to verify an HTMX application, a reasonable next goal is to support just that case. A simple HTMX application with a button and a counter, which increases when the button is pressed.
- AnXMLHttpRequest implementation needs to be in place. Work is underway for that.
- An XPathEvaluator. I believe can be poly-filled to begin with.
- Event propagation. Only DOMContentLoaded and load events are emitted right now. Elements need to support more events; such as click; as well as methods to trigger them.
- This might also require proper event capture and bubbling.
And then ...
Following that, more advanced user interaction; proper form handling, e.g., input handline (e.g., pressing enter in an field submits the form. This typically also involves some kind of URL redirection; which drives the need for a history object, etc.
Integration external sites
With the ability to control the transport layer; we can provide tests with unique abilities; we can mock out external sites that the system would depend on at run-time. E.g., for a given host name, the test could provide another Go HTTP Handler simulating the behaviour.
The most obvious example is the use of external identity providers. The test could simulate the behaviour of a login flow; not having to force you to create dummy accounts in an external system, have test failures because of outages in an external system, or simply being unable to automate the process at all because of 2FA or a Captcha introduced by the identity provider.
Another use case is the use of API-heavy libraries, like map libraries, that incur a usage cost. Mock out the external site to not receive an extra bill for running your test suite.
Usability over Compatibility
Creating a 100% whatwg standards compliant implementation is quite an endeavour; I didn't fully comprehend the scope until I actually started reading parts of the whatwg specifications. The goal is to create a tool helping write tests for web applications. Full compatibility is a long-term goal; but until the project has reached some level of usability, will I start filling out the holes.
For that reason; features that are more likely to be used in actual applications are more likely to be prioritised. A feature request pointing to an actual test giving the wrong result is likely to be prioritised. A feature request for the implementation of a specific standard is likely to be rejected.
Spread the word
I believe this can be a very useful tool for many developers, so if you read this, let your coworkers know it exists. This is so far a spare time project, of which I have plenty of at the moment; but that will not be the case forever.
If you want to see this live, spread the word ...
Perhaps you would even sponsor this? Do you have a large company building web applications in Go? Feel free to contact me.
Find the project here: https://github.com/stroiman/go-dom
-
Kudos if you managed to catch the homage to a popular BBC radio play. ↩
-
This is based on personal experience. Doing TDD right will speed you up due to the fast feedback cycle. The overhead of a real browser tends to make you write tests after the production code; loosing their benefit of the feedback loop a fast test suite gives you. ↩
-
Groundwork had already been laid by the v8go project. However; not all features of v8 are exposed to Go code; including necessary features for embedding native objects. I was able to add those in a separate fork; which is still WIP. ↩
The above is the detailed content of Go-DOM - major milestone. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
What are the vulnerabilities of Debian OpenSSL
Apr 02, 2025 am 07:30 AM
OpenSSL, as an open source library widely used in secure communications, provides encryption algorithms, keys and certificate management functions. However, there are some known security vulnerabilities in its historical version, some of which are extremely harmful. This article will focus on common vulnerabilities and response measures for OpenSSL in Debian systems. DebianOpenSSL known vulnerabilities: OpenSSL has experienced several serious vulnerabilities, such as: Heart Bleeding Vulnerability (CVE-2014-0160): This vulnerability affects OpenSSL 1.0.1 to 1.0.1f and 1.0.2 to 1.0.2 beta versions. An attacker can use this vulnerability to unauthorized read sensitive information on the server, including encryption keys, etc.
Transforming from front-end to back-end development, is it more promising to learn Java or Golang?
Apr 02, 2025 am 09:12 AM
Backend learning path: The exploration journey from front-end to back-end As a back-end beginner who transforms from front-end development, you already have the foundation of nodejs,...
What is the problem with Queue thread in Go's crawler Colly?
Apr 02, 2025 pm 02:09 PM
Queue threading problem in Go crawler Colly explores the problem of using the Colly crawler library in Go language, developers often encounter problems with threads and request queues. �...
What libraries are used for floating point number operations in Go?
Apr 02, 2025 pm 02:06 PM
The library used for floating-point number operation in Go language introduces how to ensure the accuracy is...
How to specify the database associated with the model in Beego ORM?
Apr 02, 2025 pm 03:54 PM
Under the BeegoORM framework, how to specify the database associated with the model? Many Beego projects require multiple databases to be operated simultaneously. When using Beego...
In Go, why does printing strings with Println and string() functions have different effects?
Apr 02, 2025 pm 02:03 PM
The difference between string printing in Go language: The difference in the effect of using Println and string() functions is in Go...
How to solve the user_id type conversion problem when using Redis Stream to implement message queues in Go language?
Apr 02, 2025 pm 04:54 PM
The problem of using RedisStream to implement message queues in Go language is using Go language and Redis...
What should I do if the custom structure labels in GoLand are not displayed?
Apr 02, 2025 pm 05:09 PM
What should I do if the custom structure labels in GoLand are not displayed? When using GoLand for Go language development, many developers will encounter custom structure tags...
